System for detecting an omitted event during a surgical procedure

ABSTRACT

Systems and methods for enabling determination and notification of an omitted event in a surgical procedure are disclosed. A system may include at least one processor configured to implement a method including accessing frames of video captured during a specific surgical procedure and accessing stored data identifying a recommended sequence of events for the surgical procedure. The method may include comparing the accessed frames with the recommended sequence of events to identify an indication of a deviation between the specific surgical procedure and the recommended sequence of events for the surgical procedure. The method may include determining a name of an intraoperative surgical event associated with the deviation and providing a notification of the deviation including the name of the intraoperative surgical event associated with the deviation.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation U.S. patent application Ser. No.16/983,379, filed Aug. 3, 2020, which is a continuation of U.S. patentapplication Ser. No. 16/796,304, now U.S. Pat. No. 10,729,502, issued onAug. 4, 2020, which is based on and claims benefit of priority of U.S.Provisional Patent Application No. 62/967,283, filed Jan. 29, 2020, U.S.Provisional Patent Application No. 62/960,466, filed Jan. 13, 2020, U.S.Provisional Patent Application No. 62/838,066, filed Apr. 24, 2019, U.S.Provisional Patent Application No. 62/808,512, filed Feb. 21, 2019, andU.S. Provisional Patent Application No. 62/808,500, filed Feb. 21, 2019.The contents of the foregoing applications are incorporated herein byreference in their entireties.

BACKGROUND Technical Field

The disclosed embodiments generally relate to systems and methods foranalysis of videos of surgical procedures.

Background Information

When preparing for a surgical procedure, it may be beneficial for asurgeon to view video footage depicting certain surgical events,including events that may have certain characteristics. In addition,during a surgical procedure, it may be helpful to capture and analyzevideos to provide various types of decision support to surgeons.Further, it may be helpful analyze surgical videos to facilitatepostoperative activity.

Therefore, there is a need for unconventional approaches thatefficiently and effectively analyze surgical videos to enable a surgeonto view surgical events, provide decision support, and/or facilitatepostoperative activity.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for analysis of surgical videos. The disclosed systems andmethods may be implemented using a combination of conventional hardwareand software as well as specialized hardware and software, such as amachine constructed and/or programmed specifically for performingfunctions associated with the disclosed method steps. Consistent withother disclosed embodiments, non-transitory computer-readable storagemedia may store program instructions, which are executable by at leastone processing device and perform any of the steps and/or methodsdescribed herein.

Consistent with disclosed embodiments, systems, methods, and computerreadable media related to reviewing surgical video are disclosed. Theembodiments may include accessing at least one video of a surgicalprocedure and causing the at least one video to be output for display.The embodiments may further include overlaying, on the at least onevideo outputted for display, a surgical timeline. The surgical timelinemay include markers identifying at least one of a surgical phase, anintraoperative surgical event, and a decision making junction. Thesurgical timeline may enable a surgeon, while viewing playback of the atleast one video to select one or more markers on the surgical timeline,and thereby cause a display of the video to skip to a locationassociated with the selected marker.

In one embodiment, the one or more markers may include a decision makingjunction marker corresponding to a decision making junction of thesurgical procedure. The selection of the decision making junction markermay enable the surgeon to view two or more alternative video clips fromtwo or more corresponding other surgical procedures. Further, the two ormore video clips may present differing conduct. In another embodiment,the selection of the decision making junction marker may cause a displayof one or more alternative possible decisions related to the selecteddecision making junction marker.

Consistent with disclosed embodiments, systems, methods, and computerreadable media related to video indexing are disclosed. The videoindexing may include accessing video footage to be indexed, includingfootage of a particular surgical procedure, which may be analyzed toidentify a video footage location associated with a surgical phase ofthe particular surgical procedure. A phase tag may be generated and maybe associated with the video footage location. The video indexing mayinclude analyzing the video footage to identify an event location of aparticular intraoperative surgical event within the surgical phase andassociating an event tag with the event location of the particularintraoperative surgical event. Further, an event characteristicassociated with the particular intraoperative surgical event may bestored.

The video indexing may further include associating at least a portion ofthe video footage of the particular surgical procedure with the phasetag, the event tag, and the event characteristic in a data structurethat contains additional video footage of other surgical procedures. Thedata structure may also include respective phase tags, respective eventtags, and respective event characteristics associated with one or moreof the other surgical procedures. A user may be enabled to access thedata structure through selection of a selected phase tag, a selectedevent tag, and a selected event characteristic of video footage fordisplay. Then, a lookup in the data structure of surgical video footagematching the at least one selected phase tag, selected event tag, andselected event characteristic may be performed to identify a matchingsubset of stored video footage. The matching subset of stored videofootage may be displayed to the user, thereby enabling the user to viewsurgical footage of at least one intraoperative surgical event sharingthe selected event characteristic, while omitting playback of videofootage lacking the selected event characteristic.

Consistent with disclosed embodiments, systems, methods, and computerreadable media related to generating surgical summary footage aredisclosed. The embodiments may include accessing particular surgicalfootage containing a first group of frames associated with at least oneintraoperative surgical event and a second group of frames notassociated with surgical activity. The embodiments may further includeaccessing historical data associated with historical surgical footage ofprior surgical procedures, wherein the historical data includesinformation that distinguishes portions of the historical surgicalfootage into frames associated with intraoperative surgical events andframes not associated with surgical activity. The first group of framesin the particular surgical footage may be distinguished from the secondgroup of frames based on the information of the historical data. Uponrequest of a user, an aggregate of the first group of frames of theparticular surgical footage may be presented to the user, whereas thesecond group of frames may be omitted from presentation to the user.

In some embodiments, the disclosed embodiments may further includeanalyzing the particular surgical footage to identify a surgical outcomeand a respective cause of the surgical outcome. The identifying may bebased on the historical outcome data and respective historical causedata. An outcome set of frames in the particular surgical footage may bedetected based on the analyzing. The outcome set of frames may be withinan outcome phase of the surgical procedure. Further, based on theanalyzing, a cause set of frames in the particular surgical footage maybe detected. The cause set of frames may be within a cause phase of thesurgical procedure remote in time from the outcome phase, while anintermediate set of frames may be within an intermediate phaseinterposed between the cause set of frames and the outcome set offrames. A cause-effect summary of the surgical footage may then begenerated, wherein the cause-effect summary includes the cause set offrames and the outcome set of frames and omits the intermediate set offrames. The aggregate of the first group of frames presented to the usermay include the cause-effect summary.

Consistent with disclosed embodiments, systems, methods, and computerreadable media related to surgical preparation are disclosed. Theembodiments may include accessing a repository of a plurality of sets ofsurgical video footage reflecting a plurality of surgical proceduresperformed on differing patients and including intraoperative surgicalevents, surgical outcomes, patient characteristics, surgeoncharacteristics, and intraoperative surgical event characteristics. Themethods may further include enabling a surgeon preparing for acontemplated surgical procedure to input case-specific informationcorresponding to the contemplated surgical procedure. The case-specificinformation may be compared with data associated with the plurality ofsets of surgical video footage to identify a group of intraoperativeevents likely to be encountered during the contemplated surgicalprocedure. Further, the case-specific information and the identifiedgroup of intraoperative events likely to be encountered may be used toidentify specific frames in specific sets of the plurality of sets ofsurgical video footage corresponding to the identified group ofintraoperative events. The identified specific frames may include framesfrom the plurality of surgical procedures performed on differingpatients.

The embodiments may further include determining that a first set and asecond set of video footage from differing patients contain framesassociated with intraoperative events sharing a common characteristicand omitting an inclusion of the second set from a compilation to bepresented to the surgeon and including the first set in the compilationto be presented to the surgeon. Finally, the embodiments may includeenabling the surgeon to view a presentation including the compilationcontaining frames from the differing surgical procedures performed ondiffering patients.

Consistent with disclosed embodiments, systems, methods, and computerreadable media related to analyzing complexity of surgical footage aredisclosed. The embodiments may include analyzing frames of the surgicalfootage to identify in a first set of frames an anatomical structure.The disclosed embodiments may further include accessing first historicaldata. The first historical data may be based on an analysis of firstframe data captured from a first group of prior surgical procedures. Thefirst set of frames may be analyzed using the first historical data andusing the identified anatomical structure to determine a first surgicalcomplexity level associated with the first set of frames.

Some embodiments may further include analyzing frames of the surgicalfootage to identify in a second set of frames a medical tool, theanatomical structure, and an interaction between the medical tool andthe anatomical structure. The disclosed embodiments may includeaccessing second historical data, the second historical data being basedon an analysis of a second frame data captured from a second group ofprior surgical procedures. The second set of frames may be analyzedusing the second historical data and using the identified interaction todetermine a second surgical complexity level associated with the secondset of frames.

The embodiments may further include tagging the first set of frames withthe first surgical complexity level, tagging the second set of frameswith the second surgical complexity level; and generating a datastructure including the first set of frames with the first tag and thesecond set of frames with the second tag. The generated data structuremay enable a surgeon to select the second surgical complexity level, andthereby cause the second set of frames to be displayed, while omitting adisplay of the first set of frames.

Consistent with disclosed embodiments, systems, methods, andcomputer-readable media for enabling adjustments of an operating roomschedule are disclosed. Adjusting the operating room schedule mayinclude receiving from an image sensor positioned in a surgicaloperating room, visual data tracking an ongoing surgical procedure,accessing a data structure containing historical surgical data, andanalyzing the visual data of the ongoing surgical procedure and thehistorical surgical data to determine an estimated time of completion ofthe ongoing surgical procedure. Adjusting the operating room schedulemay further include accessing a schedule for the surgical operatingroom. The schedule may include a scheduled time associated withcompletion of the ongoing surgical procedure. Further, adjusting theoperating room schedule may include calculating, based on the estimatedtime of completion of the ongoing surgical procedure, whether anexpected time of completion is likely to result in a variance from thescheduled time associated with the completion, and outputting anotification upon calculation of the variance, to thereby enablesubsequent users of the surgical operating room to adjust theirschedules accordingly.

Consistent with disclosed embodiments, systems, methods, and computerreadable media for analyzing surgical images to determine insurancereimbursement are disclosed. The operations for analyzing surgicalimages to determine insurance reimbursement may include accessing videoframes captured during a surgical procedure on a patient, analyzing thevideo frames captured during the surgical procedure to identify in thevideo frames at least one medical instrument, at least one anatomicalstructure, and at least one interaction between the at least one medicalinstrument and the at least one anatomical structure, and accessing adatabase of reimbursement codes correlated to medical instruments,anatomical structures, and interactions between medical instruments andanatomical structures. The operations may further include comparing theidentified at least one interaction between the at least one medicalinstrument and the at least one anatomical structure with information inthe database of reimbursement codes to determine at least onereimbursement code associated with the surgical procedure.

Consistent with disclosed embodiments, systems, methods, and computerreadable media for populating a post-operative report of a surgicalprocedure are disclosed. The operations for populating a post-operativereport of a surgical procedure may include receiving an input of apatient identifier, receiving an input of an identifier of a health careprovider, and receiving an input of surgical footage of a surgicalprocedure performed on the patient by the health care provider. Theoperations may further include analyzing a plurality of frames of thesurgical footage to derive image-based information for populating apost-operative report of the surgical procedure, and causing the derivedimage-based information to populate the post-operative report of thesurgical procedure.

Consistent with disclosed embodiments, systems, methods, and computerreadable media for enabling determination and notification of an omittedevent in a surgical procedure are disclosed. The operations for enablingdetermination and notification of an omitted event may include accessingframes of video captured during a specific surgical procedure, accessingstored data identifying a recommended sequence of events for thesurgical procedure, comparing the accessed frames with the recommendedsequence of events to identify an indication of a deviation between thespecific surgical procedure and the recommended sequence of events forthe surgical procedure, determining a name of an intraoperative surgicalevent associated with the deviation, and providing a notification of thedeviation including the name of the intraoperative surgical eventassociated with the deviation.

Some embodiments of this disclosure include systems, methods, andcomputer readable media for providing real-time decision support forsurgical procedures. Some of such embodiments may involve at least oneprocessor. Such embodiments may involve receiving video footage of asurgical procedure performed by a surgeon on a patient in an operatingroom and accessing at least one data structure including image-relateddata characterizing surgical procedures. Thereafter the received videofootage may be analyzed using the image-related data to determine, inreal time, an existence of a surgical decision making junction. At leastone data structure may be accessed, and a correlation between an outcomeand a specific action taken at the decision making junction. Based onthe determined existence of the decision making junction and theaccessed correlation, a recommendation may be output to the surgeon toundertake the specific action or to avoid the specific action.

Embodiments of this disclosure include systems, methods, and computerreadable media for estimating contact force on an anatomical structureduring a surgical procedure disclosed. Embodiments may involvereceiving, from at least one image sensor in an operating room, imagedata of a surgical procedure, and analyzing the received image data todetermine an identity of an anatomical structure and to determine acondition of the anatomical structure as reflected in the image data. Acontact force threshold associated with the anatomical structure may beselected based on the determined condition of the anatomical structure.An actual contact force on the anatomical structure may be determinedand compared with the selected contact force threshold. Thereafter, anotification may be output based on a determination that the indicationof actual contact force exceeds the selected contact force threshold.

Some embodiments of this disclosure involve systems, methods andcomputer readable media for updating a predicted outcome during asurgical procedure. These embodiments may involve receiving, from atleast one image sensor arranged to capture images of a surgicalprocedure, image data associated with a first event during the surgicalprocedure. The embodiments may determine, based on the received imagedata associated with the first event, a predicted outcome associatedwith the surgical procedure, and may receive, from at least one imagesensor arranged to capture images of a surgical procedure, image dataassociated with a second event during the surgical procedure. Theembodiments may then determine, based on the received image dataassociated with the second event, a change in the predicted outcome,causing the predicted outcome to drop below a threshold. A recommendedremedial action may be identified and recommended based on image-relateddata on prior surgical procedures contained in a data structure.

Some embodiments of this disclosure involve systems methods, andcomputer readable media for enabling fluid leak detection duringsurgery. Embodiments may involve receiving, in real time, intracavitaryvideo of a surgical procedure. The processor may be configured toanalyze frames of the intracavitary video to determine an abnormal fluidleakage situation in the intracavitary video. The embodiments mayinstitute a remedial action when the abnormal fluid leakage situation isdetermined.

Consistent with disclosed embodiments, systems, methods, and computerreadable media for predicting post discharge risk are disclosed. Theoperations for predicting post discharge risk may include accessingframes of video captured during a specific surgical procedure on apatient, accessing stored historical data identifying intraoperativeevents and associated outcomes, analyzing the accessed frames, and basedon information obtained from the historical data, identifying in theaccessed frames at least one specific intraoperative event, determining,based on information obtained from the historical data and theidentified at least one intraoperative event, a predicted outcomeassociated with the specific surgical procedure, and outputting thepredicted outcome in a manner associating the predicted outcome with thepatient.

The forgoing summary provides just a few examples of disclosedembodiments to provide a flavor for this disclosure and is not intendedto summarize all aspects of the disclosed embodiments. Moreover, thefollowing detailed description is exemplary and explanatory only and isnot restrictive of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a perspective view of an example operating room, consistentwith disclosed embodiments.

FIG. 2 is a perspective view of cameras, consistent with disclosedembodiments.

FIG. 3 is a perspective view of an example of a surgical instrument,consistent with disclosed embodiments.

FIG. 4 illustrates an example timeline overlaid on a video of a surgicalprocedure consistent with the disclosed embodiments.

FIG. 5 is a flowchart illustrating an example process for reviewingsurgical video, consistent with the disclosed embodiments.

FIG. 6 is a schematic illustration of an example data structureconsistent with the disclosed embodiments.

FIG. 7 is a schematic illustration of an example user interface forselecting indexed video footage for display consistent with thedisclosed embodiments.

FIGS. 8A and 8B are flowcharts illustrating an example process for videoindexing consistent with the disclosed embodiments.

FIG. 9 is a flowchart illustrating an example process for distinguishinga first group of frames from a second group of frames, consistent withthe disclosed embodiments.

FIG. 10 is a flowchart illustrating an example process for generating acause-effect summary, consistent with the disclosed embodiments.

FIG. 11 is a flowchart illustrating an example process for generatingsurgical summary footage, consistent with the disclosed embodiments.

FIG. 12 is a flowchart illustrating an exemplary process for surgicalpreparation, consistent with the disclosed embodiments.

FIG. 13 is a flowchart illustrating an exemplary process for analyzingcomplexity of surgical footage, consistent with the disclosedembodiments.

FIG. 14 is a schematic illustration of an exemplary system for managingvarious data collected during a surgical procedure, and for controllingvarious sensors consistent with disclosed embodiments.

FIG. 15 is an exemplary schedule consistent with disclosed embodiments.

FIG. 16 is an exemplary form for entering information for a scheduleconsistent with disclosed embodiments.

FIG. 17A shows an exemplary data structure consistent with disclosedembodiments.

FIG. 17B shows an exemplary plot of data of historic completion timesconsistent with disclosed embodiments.

FIG. 18 shows an example of a machine-learning model consistent withdisclosed embodiments.

FIG. 19 shows an exemplary process for adjusting an operating roomschedule consistent with disclosed embodiments.

FIG. 20 is an exemplary data structure for storing correlations betweenreimbursement codes and information obtained from surgical footage,consistent with disclosed embodiments.

FIG. 21 is block diagram of an exemplary machine learning methodconsistent with disclosed embodiments.

FIG. 22 is a flow chart of an exemplary process for analyzing surgicalimages to determine insurance reimbursement, consistent with disclosedembodiments.

FIG. 23 is an example post-operative report containing fields,consistent with disclosed embodiments.

FIG. 24A is an example of a process, including structure, for populatinga post-operative report, consistent with disclosed embodiments.

FIG. 24B is another example of a process, including structure, forpopulating a post-operative report, consistent with disclosedembodiments.

FIG. 25 is a flow diagram of an exemplary process for populating apost-operative report, consistent with disclosed embodiments.

FIG. 26 is a schematic illustration of an exemplary sequence of events,consistent with disclosed embodiments.

FIG. 27 shows an exemplary comparison of a sequence of events,consistent with disclosed embodiments.

FIG. 28 shows an exemplary process of enabling determination andnotification of an omitted event, consistent with disclosed embodiments.

FIG. 29 is a flowchart illustrating an exemplary process for decisionsupport for surgical procedures, consistent with the disclosedembodiments.

FIG. 30 is a flowchart illustrating an exemplary process for estimatingcontact force on an anatomical structure during a surgical procedure,consistent with the disclosed embodiments

FIG. 31 is a flowchart illustrating an exemplary process for updating apredicted outcome during a surgical procedure, consistent with thedisclosed embodiments.

FIG. 32 is a flowchart illustrating an exemplary process for enablingfluid leak detection during surgery, consistent with the disclosedembodiments.

FIG. 32A is an exemplary graph showing a relationship betweenintraoperative events and outcomes, consistent with disclosedembodiments.

FIG. 32B is an exemplary probability distribution graph for differentevents with and without the presence of an intraoperative event,consistent with disclosed embodiments.

FIG. 33 shows exemplary probability distribution graphs for differentevents, consistent with disclosed embodiments.

FIG. 34 shows exemplary probability distribution graphs for differentevents, as a function of event characteristics, consistent withdisclosed embodiments.

FIG. 35A shows an exemplary machine-learning model, consistent withdisclosed embodiments.

FIG. 35B shows an exemplary input for a machine-learning model,consistent with disclosed embodiments.

FIG. 36 shows an exemplary process for predicting post discharge risk,consistent with disclosed embodiments.

DETAILED DESCRIPTION

Unless specifically stated otherwise, as apparent from the followingdescription, throughout the specification discussions utilizing termssuch as “processing”, “calculating”, “computing”, “determining”,“generating”, “setting”, “configuring”, “selecting”, “defining”,“applying”, “obtaining”, “monitoring”, “providing”, “identifying”,“segmenting”, “classifying”, “analyzing”, “associating”, “extracting”,“storing”, “receiving”, “transmitting”, or the like, include actionsand/or processes of a computer that manipulate and/or transform datainto other data, the data represented as physical quantities, forexample such as electronic quantities, and/or the data representingphysical objects. The terms “computer”, “processor”, “controller”,“processing unit”, “computing unit”, and “processing module” should beexpansively construed to cover any kind of electronic device, componentor unit with data processing capabilities, including, by way ofnon-limiting example, a personal computer, a wearable computer, smartglasses, a tablet, a smartphone, a server, a computing system, a cloudcomputing platform, a communication device, a processor (for example,digital signal processor (DSP), an image signal processor (ISR), amicrocontroller, a field programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a central processing unit (CPA), agraphics processing unit (GPU), a visual processing unit (VPU), and soon), possibly with embedded memory, a single core processor, a multicore processor, a core within a processor, any other electroniccomputing device, or any combination of the above.

The operations in accordance with the teachings herein may be performedby a computer specially constructed or programmed to perform thedescribed functions.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to features of“embodiments” “one case”, “some cases”, “other cases” or variantsthereof means that a particular feature, structure or characteristicdescribed may be included in at least one embodiment of the presentlydisclosed subject matter. Thus the appearance of such terms does notnecessarily refer to the same embodiment(s). As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

Features of the presently disclosed subject matter, are, for brevity,described in the context of particular embodiments. However, it is to beunderstood that features described in connection with one embodiment arealso applicable to other embodiments. Likewise, features described inthe context of a specific combination may be considered separateembodiments, either alone or in a context other than the specificcombination.

In embodiments of the presently disclosed subject matter, one or morestages illustrated in the figures may be executed in a different orderand/or one or more groups of stages may be executed simultaneously andvice versa. The figures illustrate a general schematic of the systemarchitecture in accordance embodiments of the presently disclosedsubject matter. Each module in the figures can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in the figuresmay be centralized in one location or dispersed over more than onelocation.

Examples of the presently disclosed subject matter are not limited inapplication to the details of construction and the arrangement of thecomponents set forth in the following description or illustrated in thedrawings. The subject matter may be practiced or carried out in variousways. Also, it is to be understood that the phraseology and terminologyemployed herein is for the purpose of description and should not beregarded as limiting.

In this document, an element of a drawing that is not described withinthe scope of the drawing and is labeled with a numeral that has beendescribed in a previous drawing may have the same use and description asin the previous drawings.

The drawings in this document may not be to any scale. Different figuresmay use different scales and different scales can be used even withinthe same drawing, for example different scales for different views ofthe same object or different scales for the two adjacent objects.

Consistent with disclosed embodiments, “at least one processor” mayconstitute any physical device or group of devices having electriccircuitry that performs a logic operation on an input or inputs. Forexample, the at least one processor may include one or more integratedcircuits (IC), including application-specific integrated circuit (ASIC),microchips, microcontrollers, microprocessors, all or part of a centralprocessing unit (CPU), graphics processing unit (GPU), digital signalprocessor (DSP), field-programmable gate array (FPGA), server, virtualserver, or other circuits suitable for executing instructions orperforming logic operations. The instructions executed by at least oneprocessor may, for example, be pre-loaded into a memory integrated withor embedded into the controller or may be stored in a separate memory.The memory may include a Random Access Memory (RAM), a Read-Only Memory(ROM), a hard disk, an optical disk, a magnetic medium, a flash memory,other permanent, fixed, or volatile memory, or any other mechanismcapable of storing instructions. In some embodiments, the at least oneprocessor may include more than one processor. Each processor may have asimilar construction or the processors may be of differing constructionsthat are electrically connected or disconnected from each other. Forexample, the processors may be separate circuits or integrated in asingle circuit. When more than one processor is used, the processors maybe configured to operate independently or collaboratively. Theprocessors may be coupled electrically, magnetically, optically,acoustically, mechanically or by other means that permit them tointeract.

Disclosed embodiments may include and/or access a data structure. A datastructure consistent with the present disclosure may include anycollection of data values and relationships among them. The data may bestored linearly, horizontally, hierarchically, relationally,non-relationally, uni-dimensionally, multidimensionally, operationally,in an ordered manner, in an unordered manner, in an object-orientedmanner, in a centralized manner, in a decentralized manner, in adistributed manner, in a custom manner, or in any manner enabling dataaccess. By way of non-limiting examples, data structures may include anarray, an associative array, a linked list, a binary tree, a balancedtree, a heap, a stack, a queue, a set, a hash table, a record, a taggedunion, ER model, and a graph. For example, a data structure may includean XML database, an RDBMS database, an SQL database or NoSQLalternatives for data storage/search such as, for example, MongoDB,Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk,Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A datastructure may be a component of the disclosed system or a remotecomputing component (e.g., a cloud-based data structure). Data in thedata structure may be stored in contiguous or non-contiguous memory.Moreover, a data structure, as used herein, does not require informationto be co-located. It may be distributed across multiple servers, forexample, that may be owned or operated by the same or differententities. Thus, the term “data structure” as used herein in the singularis inclusive of plural data structures.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recursive neural network algorithms, linear machine learning models,non-linear machine learning models, ensemble algorithms, and so forth.For example, a trained machine learning algorithm may comprise aninference model, such as a predictive model, a classification model, aregression model, a clustering model, a segmentation model, anartificial neural network (such as a deep neural network, aconvolutional neural network, a recursive neural network, etc.), arandom forest, a support vector machine, and so forth. In some examples,the training examples may include example inputs together with thedesired outputs corresponding to the example inputs. Further, in someexamples, training machine learning algorithms using the trainingexamples may generate a trained machine learning algorithm, and thetrained machine learning algorithm may be used to estimate outputs forinputs not included in the training examples. In some examples,engineers, scientists, processes and machines that train machinelearning algorithms may further use validation examples and/or testexamples. For example, validation examples and/or test examples mayinclude example inputs together with the desired outputs correspondingto the example inputs, a trained machine learning algorithm and/or anintermediately trained machine learning algorithm may be used toestimate outputs for the example inputs of the validation examplesand/or test examples, the estimated outputs may be compared to thecorresponding desired outputs, and the trained machine learningalgorithm and/or the intermediately trained machine learning algorithmmay be evaluated based on a result of the comparison. In some examples,a machine learning algorithm may have parameters and hyper parameters,where the hyper parameters are set manually by a person or automaticallyby an process external to the machine learning algorithm (such as ahyper parameter search algorithm), and the parameters of the machinelearning algorithm are set by the machine learning algorithm accordingto the training examples. In some implementations, the hyper-parametersare set according to the training examples and the validation examples,and the parameters are set according to the training examples and theselected hyper-parameters.

In some embodiments, trained machine learning algorithms (also referredto as trained machine learning models in the present disclosure) may beused to analyze inputs and generate outputs, for example in the casesdescribed below. In some examples, a trained machine learning algorithmmay be used as an inference model that when provided with an inputgenerates an inferred output. For example, a trained machine learningalgorithm may include a classification algorithm, the input may includea sample, and the inferred output may include a classification of thesample (such as an inferred label, an inferred tag, and so forth). Inanother example, a trained machine learning algorithm may include aregression model, the input may include a sample, and the inferredoutput may include an inferred value for the sample. In yet anotherexample, a trained machine learning algorithm may include a clusteringmodel, the input may include a sample, and the inferred output mayinclude an assignment of the sample to at least one cluster. In anadditional example, a trained machine learning algorithm may include aclassification algorithm, the input may include an image, and theinferred output may include a classification of an item depicted in theimage. In yet another example, a trained machine learning algorithm mayinclude a regression model, the input may include an image, and theinferred output may include an inferred value for an item depicted inthe image (such as an estimated property of the item, such as size,volume, age of a person depicted in the image, cost of a productdepicted in the image, and so forth). In an additional example, atrained machine learning algorithm may include an image segmentationmodel, the input may include an image, and the inferred output mayinclude a segmentation of the image. In yet another example, a trainedmachine learning algorithm may include an object detector, the input mayinclude an image, and the inferred output may include one or moredetected objects in the image and/or one or more locations of objectswithin the image. In some examples, the trained machine learningalgorithm may include one or more formulas and/or one or more functionsand/or one or more rules and/or one or more procedures, the input may beused as input to the formulas and/or functions and/or rules and/orprocedures, and the inferred output may be based on the outputs of theformulas and/or functions and/or rules and/or procedures (for example,selecting one of the outputs of the formulas and/or functions and/orrules and/or procedures, using a statistical measure of the outputs ofthe formulas and/or functions and/or rules and/or procedures, and soforth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs. Some non-limitingexamples of such artificial neural networks may comprise shallowartificial neural networks, deep artificial neural networks, feedbackartificial neural networks, feed forward artificial neural networks,autoencoder artificial neural networks, probabilistic artificial neuralnetworks, time delay artificial neural networks, convolutionalartificial neural networks, recurrent artificial neural networks, longshort term memory artificial neural networks, and so forth. In someexamples, an artificial neural network may be configured manually. Forexample, a structure of the artificial neural network may be selectedmanually, a type of an artificial neuron of the artificial neuralnetwork may be selected manually, a parameter of the artificial neuralnetwork (such as a parameter of an artificial neuron of the artificialneural network) may be selected manually, and so forth. In someexamples, an artificial neural network may be configured using a machinelearning algorithm. For example, a user may select hyper-parameters forthe an artificial neural network and/or the machine learning algorithm,and the machine learning algorithm may use the hyper-parameters andtraining examples to determine the parameters of the artificial neuralnetwork, for example using back propagation, using gradient descent,using stochastic gradient descent, using mini-batch gradient descent,and so forth. In some examples, an artificial neural network may becreated from two or more other artificial neural networks by combiningthe two or more other artificial neural networks into a singleartificial neural network.

In some embodiments, analyzing image data (for example by the methods,steps and modules described herein) may comprise analyzing the imagedata to obtain a preprocessed image data, and subsequently analyzing theimage data and/or the preprocessed image data to obtain the desiredoutcome. Some non-limiting examples of such image data may include oneor more images, videos, frames, footages, 2D image data, 3D image data,and so forth. One of ordinary skill in the art will recognize that thefollowings are examples, and that the image data may be preprocessedusing other kinds of preprocessing methods. In some examples, the imagedata may be preprocessed by transforming the image data using atransformation function to obtain a transformed image data, and thepreprocessed image data may comprise the transformed image data. Forexample, the transformed image data may comprise one or moreconvolutions of the image data. For example, the transformation functionmay comprise one or more image filters, such as low-pass filters,high-pass filters, band-pass filters, all-pass filters, and so forth. Insome examples, the transformation function may comprise a nonlinearfunction. In some examples, the image data may be preprocessed bysmoothing at least parts of the image data, for example using Gaussianconvolution, using a median filter, and so forth. In some examples, theimage data may be preprocessed to obtain a different representation ofthe image data. For example, the preprocessed image data may comprise: arepresentation of at least part of the image data in a frequency domain;a Discrete Fourier Transform of at least part of the image data; aDiscrete Wavelet Transform of at least part of the image data; atime/frequency representation of at least part of the image data; arepresentation of at least part of the image data in a lower dimension;a lossy representation of at least part of the image data; a losslessrepresentation of at least part of the image data; a time ordered seriesof any of the above; any combination of the above; and so forth. In someexamples, the image data may be preprocessed to extract edges, and thepreprocessed image data may comprise information based on and/or relatedto the extracted edges. In some examples, the image data may bepreprocessed to extract image features from the image data. Somenon-limiting examples of such image features may comprise informationbased on and/or related to edges; corners; blobs; ridges; ScaleInvariant Feature Transform (SIFT) features; temporal features; and soforth.

In some embodiments, analyzing image data (for example, by the methods,steps and modules described herein) may comprise analyzing the imagedata and/or the preprocessed image data using one or more rules,functions, procedures, artificial neural networks, object detectionalgorithms, face detection algorithms, visual event detectionalgorithms, action detection algorithms, motion detection algorithms,background subtraction algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth.

In some embodiments, analyzing image data (for example, by the methods,steps and modules described herein) may comprise analyzing pixels,voxels, point cloud, range data, etc. included in the image data.

FIG. 1 shows an example operating room 101, consistent with disclosedembodiments. A patient 143 is illustrated on an operating table 141.Room 101 may include audio sensors, video/image sensors, chemicalsensors, and other sensors, as well as various light sources (e.g.,light source 119 is shown in FIG. 1 ) for facilitating the capture ofvideo and audio data, as well as data from other sensors, during thesurgical procedure. For example, room 101 may include one or moremicrophones (e.g., audio sensor 111, as shown in FIG. 1 ), severalcameras (e.g., overhead cameras 115, 121, and 123, and a tablesidecamera 125) for capturing video/image data during surgery. While some ofthe cameras (e.g., cameras 115, 123 and 125) may capture video/imagedata of operating table 141 (e.g., the cameras may capture thevideo/image data at a location 127 of a body of patient 143 on which asurgical procedure is performed), camera 121 may capture video/imagedata of other parts of operating room 101. For instance, camera 121 maycapture video/image data of a surgeon 131 performing the surgery. Insome cases, cameras may capture video/image data associated withsurgical team personnel, such as an anesthesiologist, nurses, surgicaltech and the like located in operating room 101. Additionally, operatingroom cameras may capture video/image data associated with medicalequipment located in the room.

In various embodiments, one or more of cameras 115, 121, 123 and 125 maybe movable. For example, as shown in FIG. 1 , camera 115 may be rotatedas indicated by arrows 135A showing a pitch direction, and arrows 135Bshowing a yaw direction for camera 115. In various embodiments, pitchand yaw angles of cameras (e.g., camera 115) may be electronicallycontrolled such that camera 115 points at a region-of-interest (ROI), ofwhich video/image data needs to be captured. For example, camera 115 maybe configured to track a surgical instrument (also referred to as asurgical tool) within location 127, an anatomical structure, a hand ofsurgeon 131, an incision, a movement of anatomical structure, and thelike. In various embodiments, camera 115 may be equipped with a laser137 (e.g., an infrared laser) for precision tracking. In some cases,camera 115 may be tracked automatically via a computer-based cameracontrol application that uses an image recognition algorithm forpositioning the camera to capture video/image data of a ROI. Forexample, the camera control application may identify an anatomicalstructure, identify a surgical tool, hand of a surgeon, bleeding,motion, and the like at a particular location within the anatomicalstructure, and track that location with camera 115 by rotating camera115 by appropriate yaw and pitch angles. In some embodiments, the cameracontrol application may control positions (i.e., yaw and pitch angles)of various cameras 115, 121, 123 and 125 to capture video/image datefrom different ROIs during a surgical procedure. Additionally oralternatively, a human operator may control the position of variouscameras 115, 121, 123 and 125, and/or the human operator may supervisethe camera control application in controlling the position of thecameras.

Cameras 115, 121, 123 and 125 may further include zoom lenses forfocusing in on and magnifying one or more ROIs. In an exampleembodiment, camera 115 may include a zoom lens 138 for zooming closelyto a ROI (e.g., a surgical tool in the proximity of an anatomicalstructure). Camera 121 may include a zoom lens 139 for capturingvideo/image data from a larger area around the ROI. For example, camera121 may capture video/image data for the entire location 127. In someembodiments, video/image data obtained from camera 121 may be analyzedto identify a ROI during the surgical procedure, and the camera controlapplication may be configured to cause camera 115 to zoom towards theROI identified by camera 121.

In various embodiments, the camera control application may be configuredto coordinate the position, focus, and magnification of various camerasduring a surgical procedure. For example, the camera control applicationmay direct camera 115 to track an anatomical structure and may directcamera 121 and 125 to track a surgical instrument. Cameras 121 and 125may track the same ROI (e.g., a surgical instrument) from different viewangles. For example, video/image data obtained from different viewangles may be used to determine the position of the surgical instrumentrelative to a surface of the anatomical structure, to determine acondition of an anatomical structure, to determine pressure applied toan anatomical structure, or to determine any other information wheremultiple viewing angles may be beneficial. By way of another example,bleeding may be detected by one camera, and one or more other camerasmay be used to identify the source of the bleeding.

In various embodiments, control of position, orientation, settings,and/or zoom of cameras 115, 121, 123 and 125 may be rule-based andfollow an algorithm developed for a given surgical procedure. Forexample, the camera control application may be configured to directcamera 115 to track a surgical instrument, to direct camera 121 tolocation 127, to direct camera 123 to track the motion of the surgeon'shands, and to direct camera 125 to an anatomical structure. Thealgorithm may include any suitable logical statements determiningposition, orientation, settings and/or zoom for cameras 115, 121, 123and 125 depending on various events during the surgical procedure. Forexample, the algorithm may direct at least one camera to a region of ananatomical structure that develops bleeding during the procedure. Somenon-limiting examples of settings of cameras 115, 121, 123 and 125 thatmay be controlled (for example by the camera control application) mayinclude image pixel resolution, frame rate, image and/or colorcorrection and/or enhancement algorithms, zoom, position, orientation,aspect ratio, shutter speed, aperture, focus, and so forth.

In various cases, when a camera (e.g., camera 115) tracks a moving ordeforming object (e.g., when camera 115 tracks a moving surgicalinstrument, or a moving/pulsating anatomical structure), a cameracontrol application may determine a maximum allowable zoom for camera115, such that the moving or deforming object does not escape a field ofview of the camera. In an example embodiment, the camera controlapplication may initially select the first zoom for camera 115, evaluatewhether the moving or deforming object escapes the field of view of thecamera, and adjust the zoom of the camera as necessary to prevent themoving or deforming object from escaping the field of view of thecamera. In various embodiments, the camera zoom may be readjusted basedon a direction and a speed of the moving or deforming object.

In various embodiments, one or more image sensors may include movingcameras 115, 121, 123 and 125. Cameras 115, 121, 123 and 125 may be usedfor determining sizes of anatomical structures and determining distancesbetween different ROIs, for example using triangulation. For example,FIG. 2 shows exemplary cameras 115 (115 View 1, as shown in FIG. 2 ) and121 supported by movable elements such that the distance between the twocameras is D₁, as shown in FIG. 2 . Both cameras point at ROI 223. Byknowing the positions of cameras 115 and 121 and the direction of anobject relative to the cameras (e.g., by knowing angles A₁ and A₂, asshown in FIG. 2 , for example based on correspondences between pixelsdepicting the same object or the same real-world point in the imagescaptured by 115 and 121), distances D₂ and D₃ may be calculated using,for example, the law of sines and the known distance between the twocameras D₁. In an example embodiment, when camera 115 (115, View 2)rotates by a small angle A₃ (measured in radians), to point at ROI 225,the distance between ROI 223 and ROI 225 may be approximated (for smallangles A₃) by A₃D₂. More accuracy may be obtained using anothertriangulation process. Knowing distances between ROI 223 and 225 allowsdetermining a length scale for an anatomical structure. Further,distances between various points of the anatomical structure, anddistances from the various points to one or more cameras may be measuredto determine a point-cloud representing a surface of the anatomicalstructure. Such a point-cloud may be used to reconstruct athree-dimensional model of the anatomical structure. Further, distancesbetween one or more surgical instruments and different points of theanatomical structure may be measured to determine proper locations ofthe one or more surgical instruments in the proximity of the anatomicalstructure. In some other examples, one or more of cameras 115, 121, 123and 125 may include a 3D camera (such as a stereo camera, an activestereo camera, a Time of Flight camera, a Light Detector and Rangingcamera, etc.), and actual and/or relative locations and/or sizes ofobjects within operating room 101, and/or actual distances betweenobjects, may be determined based on the 3D information captured by the3D camera.

Returning to FIG. 1 , light sources (e.g., light source 119) may also bemovable to track one or more ROIs. In an example embodiment, lightsource 119 may be rotated by yaw and pitch angles, and in some cases,may extend towards to or away from a ROI (e.g., location 127). In somecases, light source 119 may include one or more optical elements (e.g.,lenses, flat or curved mirrors, and the like) to focus light on the ROI.In some cases, light source 119 may be configured to control the colorof the light (e.g., the color of the light may include different typesof white light, a light with a selected spectrum, and the like). In anexample embodiment, light 119 may be configured such that the spectrumand intensity of the light may vary over a surface of an anatomicstructure illuminated by the light. For example, in some cases, light119 may include infrared wavelengths which may result in warming of atleast some portions of the surface of the anatomic structure.

In some embodiments, the operating room may include sensors embedded invarious components depicted or not depicted in FIG. 1 . Examples of suchsensors may include: audio sensors; image sensors; motion sensors;positioning sensors; chemical sensors; temperature sensors; barometers;pressure sensors; proximity sensors; electrical impedance sensors;electrical voltage sensors; electrical current sensors; or any otherdetector capable of providing feedback on the environment or a surgicalprocedure, including, for example, any kind of medical or physiologicalsensor configured to monitor patient 143.

In some embodiments, audio sensor 111 may include one or more audiosensors configured to capture audio by converting sounds to digitalinformation (e.g., audio sensors 121).

In various embodiments, temperature sensors may include infrared cameras(e.g., an infrared camera 117 is shown in FIG. 1 ) for thermal imaging.Infrared camera 117 may allow measurements of the surface temperature ofan anatomic structure at different points of the structure. Similar tovisible cameras D115, 121, 123 and 125, infrared camera 117 may berotated using yaw or pitch angles. Additionally or alternatively, camera117 may include an image sensor configured to capture image from anylight spectrum, include infrared image sensor, hyper-spectral imagesensors, and so forth.

FIG. 1 includes a display screen 113 that may show views from differentcameras 115, 121, 123 and 125, as well as other information. Forexample, display screen 113 may show a zoomed-in image of a tip of asurgical instrument and a surrounding tissue of an anatomical structurein proximity to the surgical instrument.

FIG. 3 shows an example embodiment of a surgical instrument 301 that mayinclude multiple sensors and light-emitting sources. Consistent with thepresent embodiments, a surgical instrument may refer to a medicaldevice, a medical instrument, an electrical or mechanical tool, asurgical tool, a diagnostic tool, and/or any other instrumentality thatmay be used during a surgery. As shown, instrument 301 may includecameras 311A and 311B, light sources 313A and 313B as well as tips 323Aand 323B for contacting tissue 331. Cameras 311A and 311B may beconnected via data connection 319A and 319B to a data transmittingdevice 321. In an example embodiment, device 321 may transmit data to adata-receiving device using a wireless communication or using a wiredcommunication. In an example embodiment, device 321 may use WiFi,Bluetooth, NFC communication, inductive communication, or any othersuitable wireless communication for transmitting data to adata-receiving device. The data-receiving device may include any form ofreceiver capable of receiving data transmissions. Additionally oralternatively, device 321 may use optical signals to transmit data tothe data-receiving device (e.g., device 321 may use optical signalstransmitted through the air or via optical fiber). In some embodiments,device 301 may include local memory for storing at least some of thedata received from sensors 311A and 311B. Additionally, device 301 mayinclude a processor for compressing video/image data before transmittingthe data to the data-receiving device.

In various embodiments, for example when device 301 is wireless, it mayinclude an internal power source (e.g., a battery, a rechargeablebattery, and the like) and/or a port for recharging the battery, anindicator for indicating the amount of power remaining for the powersource, and one or more input controls (e.g., buttons) for controllingthe operation of device 301. In some embodiments, control of device 301may be accomplished using an external device (e.g., a smartphone,tablet, smart glasses) communicating with device 301 via any suitableconnection (e.g., WiFi, Bluetooth, and the like). In an exampleembodiment, input controls for device 301 may be used to control variousparameters of sensors or light sources. For example, input controls maybe used to dim/brighten light sources 313A and 313B, move the lightsources for cases when the light sources may be moved (e.g., the lightsources may be rotated using yaw and pitch angles), control the color ofthe light sources, control the focusing of the light sources, controlthe motion of cameras 311A and 311B for cases when the cameras may bemoved (e.g., the cameras may be rotated using yaw and pitch angles),control the zoom and/or capturing parameters for cameras 311A and 311B,or change any other suitable parameters of cameras 311A-311B and lightsources 313A-313B. It should be noted camera 311A may have a first setof parameters and camera 311B may have a second set of parameters thatis different from the first set of parameters, and these parameters maybe selected using appropriate input controls. Similarly, light source313A may have a first set of parameters and light source 313B may have asecond set of parameters that is different from the first set ofparameters, and these parameters may be selected using appropriate inputcontrols.

Additionally, instrument 301 may be configured to measure data relatedto various properties of tissue 331 via tips 323A and 323B and transmitthe measured data to device 321. For example, tips 323A and 323B may beused to measure the electrical resistance and/or impedance of tissue331, the temperature of tissue 331, mechanical properties of tissue 331and the like. To determine elastic properties of tissue 331, forexample, tips 323A and 323B may be first separated by an angle 317 andapplied to tissue 331. The tips may be configured to move such as toreduce angle 317, and the motion of tips may result in pressure ontissue 331. Such pressure may be measured (e.g., via a piezoelectricelement 327 that may be located between a first branch 312A and a secondbranch 312B of instrument 301), and based on the change in angle 317(i.e., strain) and the measured pressure (i.e., stress), the elasticproperties of tissue 331 may be measured. Furthermore, based on angle317 distance between tips 323A and 323B may be measured, and thisdistance may be transmitted to device 321. Such distance measurementsmay be used as a length scale for various video/image data that may becaptured by various cameras 115, 121, 123 and 125, as shown in FIG. 1 .

Instrument 301 is only one example of possible surgical instrument, andother surgical instruments such as scalpels, graspers (e.g., forceps),clamps and occluders, needles, retractors, cutters, dilators, suctiontips, and tubes, sealing devices, irrigation and injection needles,scopes and probes, and the like, may include any suitable sensors andlight-emitting sources. In various cases, the type of sensors andlight-emitting sources may depend on a type of surgical instrument usedfor a surgical procedure. In various cases, these other surgicalinstruments may include a device similar to device 301, as shown in FIG.3 , for collecting and transmitting data to any suitable data-receivingdevice.

When preparing for a surgical procedure, it may be beneficial for asurgeon to review video footage of surgical procedures having similarsurgical events. It may be too time consuming, however, for a surgeon toview the entire video or to skip around to find relevant portions of thesurgical footage. Therefore, there is a need for unconventionalapproaches that efficiently and effectively enable a surgeon to view asurgical video summary that aggregates footage of relevant surgicalevents while omitting other irrelevant footage.

Aspects of this disclosure may relate to reviewing surgical video,including methods, systems, devices, and computer readable media. Aninterface may allow a surgeon to review surgical video (of their ownsurgeries, other's surgeries, or compilations) with a surgical timelinesimultaneously displayed. The timeline may include markers keyed toactivities or events that occur during a surgical procedure. Thesemarkers may allow the surgeon to skip to particular activities tothereby streamline review of the surgical procedure. In someembodiments, key decision making junction points may be marked, and thesurgeon may be permitted to view alternative actions taken at thosedecision making junction points.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In a broadest sense, the method is not limited toparticular physical and/or electronic instrumentalities, but rather maybe accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve accessing atleast one video of a surgical procedure. As described in greater detailabove, video may include any form of recorded visual media includingrecorded images and/or sound. The video may be stored as a video filesuch as an Audio Video Interleave (AVI) file, a Flash Video Format (FLV)file, QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), a WindowsMedia Video (WMV) file, a Material Exchange Format (MXF) file, or anyother suitable video file formats, for example as described above.

A surgical procedure may include any medical procedure associated withor involving manual or operative procedures on a patient's body.Surgical procedures may include cutting, abrading, suturing, or othertechniques that involve physically changing body tissues and organs.Examples of such surgical procedures are provided above. A video of asurgical procedure may include any series of still images that werecaptured during and are associated with the surgical procedure. In someembodiments, at least a portion of the surgical procedure may bedepicted in one or more of the still images included in the video. Forexample, the video of the surgical procedure may be recorded by an imagecapture device, such as a camera, in an operating room or in a cavity ofa patient. Accessing the video of the surgical procedure may includeretrieving the video from a storage device (such as one or more memoryunits, a video server, a cloud storage platform, or any other storageplatform), receiving the video from another device through acommunication device, capturing the video using image sensors, or anyother means for electronically accessing data or files.

Some aspects of the present disclosure may involve causing the at leastone video to be output for display. Outputting the at least one videomay include any process by which the video is produced, delivered, orsupplied using a computer or at least one processor. As used herein.“display” may refer to any manner in which a video may be presented to auser for playback. In some embodiments, outputting the video may includepresenting the video using a display device, such as a screen (e.g., anOLED, QLED LCD, plasma, CRT, DLPT, electronic paper, or similar displaytechnology), a light projector (e.g., a movie projector, a slideprojector), a 3D display, screen of a mobile device, electronic glassesor any other form of visual and/or audio presentation. In otherembodiments, outputting the video for display may include storing thevideo in a location that is accessible by one or more other computingdevices. Such storage locations may include a local storage (such as ahard drive of flash memory), a network location (such as a server ordatabase), a cloud computing platform, or any other accessible storagelocation. The video may be accessed from a separate computing device fordisplay on the separate computing device. In some embodiments,outputting the video may include transmitting the video to an externaldevice. For example, outputting the video for display may includetransmitting the video through a network to a user device for playbackon the user device.

Embodiments of the present disclosure may further include overlaying onthe at least one video outputted for display a surgical timeline. Asused herein, a “timeline” may refer to any depiction from which asequence of events may be tracked or demarcated. In some embodiments, atimeline may be a graphical representation of events, for example, usingan elongated bar or line representing time with markers or otherindicators of events along the bar. A timeline may also be a text-basedlist of events arranged in chronological order. A surgical timeline maybe a timeline representing events associated with a surgery. As oneexample, a surgical timeline may be a timeline of events or actions thatoccur during a surgical procedure, as described in detail above. In someembodiments, the surgical timeline may include textual informationidentifying portions of the surgical procedure. For example, thesurgical timeline may be a list of descriptions of intraoperativesurgical events or surgical phases within a surgical procedure. In otherembodiments, by hovering over or otherwise actuating graphical markerson a timeline, a descriptor associated with the marker may appear.

Overlaying the surgical timeline on the at least one video may includeany manner of displaying the surgical timeline such that it can beviewed simultaneously with the at least one video. In some embodiments,overlaying the video may include displaying the surgical timeline suchthat it at least partially overlaps the video. For example, the surgicaltimeline may be presented as a horizontal bar along a top or bottom ofthe video or a vertical bar along a side of the video. In otherembodiments, overlaying may include presenting the surgical timelinealongside the video. For example, the video may be presented on adisplay with the surgical timeline presented above, below, and/or to theside of the video. The surgical timeline may be overlaid on the videowhile the video is being played. Thus, “overlaying” as used hereinrefers more generally to simultaneous display. The simultaneous displaymay or may not be constant. For example, the overlay may appear with thevideo output before the end of the surgical procedure depicted in thedisplayed video. Or, the overlay may appear during substantially all ofthe video procedure.

FIG. 4 illustrates an example timeline 420 overlaid on a video of asurgical procedure consistent with the disclosed embodiments. The videomay be presented in a video playback region 410, which may sequentiallydisplay one or more frames of the video. In the example shown in FIG. 4, timeline 420 may be displayed as a horizontal bar representing time,with the leftmost portion of the bar representing a beginning time ofthe video and the rightmost portion of the bar representing an end time.Timeline 420 may include a position indicator 424 indicating the currentplayback position of the video relative to the timeline. Colored region422 of timeline 420 may represent the progress within timeline 420(e.g., corresponding to video that has already been viewed by the user,or to video coming before the currently presented frame). In someembodiments, position indicator 424 may be interactive, such that theuser can move to different positions within the video by moving positionindicator 424. In some embodiments, the surgical timeline may includemarkers identifying at least one of a surgical phase, an intraoperativesurgical event, and a decision making junction. For example, timeline420 may further include one or more markers 432, 434, and/or 436. Suchmarkers are described in greater detail below.

In the example shown in FIG. 4 , timeline 420 may be displayed such thatit overlaps video playback region 410, either physically, temporally, orboth. In some embodiments, timeline 420 may not be displayed at alltimes. As one example, timeline 420 may automatically switch to acollapsed or hidden view while a user is viewing the video and mayreturn to the expanded view shown in FIG. 4 when the user takes anaction to interact with timeline 420. For example, user may move a mousepointer while viewing the video, move the mouse pointer over thecollapsed timeline, move the mouse pointer to a particular region, clickor tap the video playback region, or perform any other actions that mayindicate an intent to interact with timeline 420. As discussed above,timeline 420 may be displayed in various other locations relative tovideo playback region 410, including on a top portion of video playbackregion 410, above or below video playback region 410, or within controlbar 612. In some embodiments, timeline 420 may be displayed separatelyfrom a video progress bar. For example, a separate video progress bar,including position indicator 424 and colored region 422, may bedisplayed in control bar 412 and timeline 420 may be a separate timelineof events associated with a surgical procedure. In such embodiments,timeline 420 may not have the same scale or range of time as the videoor the video progress bar. For example, the video progress bar mayrepresent the time scale and range of the video, whereas timeline 420may represent the timeframe of the surgical procedure, which may not bethe same (e.g., where the video comprises a surgical summary, asdiscussed in detail above). In some embodiments, video playback region410 may include a search icon 440, which may allow a user to search forvideo footage, for example, through user interface 700, as describedabove in reference to FIG. 7 . The surgical timeline shown in FIG. 4 isprovided by way of example only, and one skilled in the art wouldappreciate various other configurations that may be used.

Embodiments of the present disclosure may further include enabling asurgeon, while viewing playback of the at least one video to select oneor more markers on the surgical timeline, and thereby cause a display ofthe video to skip to a location associated with the selected marker. Asused herein, “playback” may include any presentation of a video in whichone or more frames of the video are displayed to the user. Typically,playback will include sequentially displaying the images to reproducemoving images and/or sounds, however playback may also include thedisplay of individual frames.

Consistent with the disclosed embodiments, a “marker” may include anyvisual indicator associated with location within the surgical timeline.As described above, the location may refer to any particular positionwithin a video. For example, the location may be a particular frame orrange of frames in the video, a particular timestamp, or any otherindicator of position within the video. Markers may be represented onthe timeline in various ways. In some embodiments, the markers may beicons or other graphic representations displayed along the timeline atvarious locations. The markers may be displayed as lines, bands, dots,geometric shapes (such as diamonds, squares, triangles, or any othershape), bubbles, or any other graphical or visual representation. Insome embodiments, the markers may be text-based. For example, themarkers may include textual information, such as a name, a description,a code, a timestamp, and so forth. In another example, the surgicaltimeline may be displayed as a list, as described above. Accordingly,the markers may include text-based titles or descriptions referring to aparticular location of the video. Markers 432, 434, and 438 are shown byway of example in FIG. 4 . The markers may be represented as calloutbubbles, including an icon indicating the type of marker associated withthe location. The markers may point to a particular point along timeline420 indicating the location in the video.

Selection of the marker may include any action by a user directedtowards a particular marker. In some embodiments, selecting the markermay include clicking on or tapping the marker through a user interface,touching the marker on a touch sensitive screen, glancing at the markerthrough smart glasses, indicating the marker through a voice interface,indicating the marker with a gesture, or undertaking any other actionthat causes the marker to be selected. Selection of the marker maythereby cause a display of the video to skip to a location associatedwith the selected marker. As used herein, skipping may includeselectively displaying a particular frame within a video. This mayinclude stopping display of a frame at a current location in the video(for example, if the video is currently playing) and displaying a frameat the location associated with the selected marker. For example, if auser clicks on or otherwise selects marker 432, as shown in FIG. 4 , aframe at the location associated with marker 432 may be displayed invideo playback region 410. In some embodiments, the video may continueplaying from that location. Position indicator 424 may move to aposition within timeline 420 associated with marker 432 and coloredregion 422 may be updated accordingly. While the present embodiment isdescribed as enabling a surgeon to select the one or more markers, it isunderstood that this is an example only, and the present disclosure isnot limited to any form of user. Various other users may view andinteract with the overlaid timeline, including a surgical technician, anurse, a physician's assistant, an anesthesiologist, a doctor, or anyother healthcare professional, as well as a patient, an insurer, amedical student, and so forth. Other examples of users are providedherein.

In accordance with embodiments of the present disclosure, the markersmay be automatically generated and included in the timeline based oninformation in the video at a given location. In some embodiments,computer analysis may be used to analyze frames of the video footage andidentify markers to include at various locations in the timeline.Computer analysis may include any form of electronic analysis using acomputing device. In some embodiments, computer analysis may includeusing one or more image recognition algorithms to identify features ofone or more frames of the video footage. Computer analysis may beperformed on individual frames, or may be performed across multipleframes, for example, to detect motion or other changes between frames.In some embodiments computer analysis may include object detectionalgorithms, such as Viola-Jones object detection, scale-invariantfeature transform (SIFT), histogram of oriented gradients (HOG)features, convolutional neural networks (CNN), or any other forms ofobject detection algorithms. Other example algorithms may include videotracking algorithms, motion detection algorithms, feature detectionalgorithms, color-based detection algorithms, texture based detectionalgorithms, shape based detection algorithms, boosting based detectionalgorithms, face detection algorithms, or any other suitable algorithmfor analyzing video frames. In one example, a machine learning model maybe trained using training examples to generate markers for videos, andthe trained machine learning model may be used to analyze the video andgenerate markers for that video. Such generated markers may includelocations within the video for the marker, type of the marker,properties of the marker, and so forth. An example of such trainingexample may include a video clip depicting at least part of a surgicalprocedure, together with a list of desired markers to be generated,possibly together with information for each desired marker, such as alocation within the video for the marker, a type of the marker,properties of the marker, and so forth.

This computer analysis may be used to identify surgical phases,intraoperative events, event characteristics, and/or other featuresappearing in the video footage. For example, in some embodiments,computer analysis may be used to identify one or more medicalinstruments used in a surgical procedure, for example as describedabove. Based on identification of the medical instrument, a particularintraoperative event may be identified at a location in the videofootage associated with the medical instrument. For example, a scalpelor other instrument may indicate that an incision is being made and amarker identifying the incision may be included in the timeline at thislocation. In some embodiments, anatomical structures may be identifiedin the video footage using the computer analysis, for example asdescribed above. For example, the disclosed methods may includeidentifying organs, tissues, fluids or other structures of the patientto determine markers to include in the timeline and their respectivelocations. In some embodiments, locations for video markers may bedetermined based on an interaction between a medical instrument and theanatomical structure, which may indicate a particular intraoperativeevent, type of surgical procedure, event characteristic, or otherinformation useful in identifying marker locations. For example, visualaction recognition algorithms may be used to analyze the video anddetect the interactions between the medical instrument and theanatomical structure. Other examples of features that may be detected invideo footage for placing markers may include, motions of a surgeon orother medical professional, patient characteristics, surgeoncharacteristics or characteristics of other medical professionals,sequences of operations being performed, timings of operations orevents, characteristics of anatomical structures, medical conditions, orany other information that may be used to identify particular surgicalprocedures, surgical phases, intraoperative events, and/or eventcharacteristics appearing in the video footage.

In some embodiments, marker locations may be identified using a trainedmachine learning model. For example, a machine learning model may betrained using training examples, each training example may include videofootage known to be associated with surgical procedures, surgicalphases, intraoperative events, and/or event characteristics, togetherwith labels indicating locations within the video footage. Using thetrained machine learning model, similar phases and events may beidentified in other video footage for the determining marker locations.Various machine learning models may be used, including a logisticregression model, a linear regression model, a regression model, arandom forest model, a K-Nearest Neighbor (KNN) model, a K-Means model,a decision tree, a cox proportional hazards regression model, a NaïveBayes model, a Support Vector Machines (SVM) model, a gradient boostingalgorithm, artificial neural networks (such as deep neural networks,convolutional neural networks, etc.) or any other form of machinelearning model or algorithm.

In some embodiments, video markers may be identified in conjunction withthe video indexing techniques discussed above. As described above, videofootage may be indexed based on surgical phases, intraoperative events,and/or event characteristics identified in the video footage. Thisinformation may be stored in a data structure, such as data structure600, as described in reference to FIG. 6 . The data structure mayinclude footage locations and/or event locations associated with phasesand events within the video footage. In some embodiments, the markersdisplayed in the timeline may correspond to these locations in thevideo. Accordingly any of the techniques or processes described abovefor indexing video footage may similarly apply to determining markerlocations for presenting in a timeline.

According to various exemplary embodiments of the present disclosure,the markers may be coded by at least one of a color or a criticalitylevel. The coding of a marker may be any indicator of a type, property,or characteristic of the marker. The coding may be useful for a user invisually determining which locations of the video may be of interest.Where the marker is coded by color, the color of the marker displayed onthe surgical timeline may indicate the property or characteristic of themarker based on a predefined color scheme. For example, the marker mayhave a different color depending on what type of intraoperative surgicalevent the marker represents. In some example embodiments, markersassociated with an incision, an excision, a resection, a ligation, agraft, or various other events may each be displayed with a differentcolor. In other embodiments, intraoperative adverse events may beassociated with one color (e.g., red), where planned events may beassociated with another color (e.g., green). In some embodiments, colorscales may be used. For example, the severity of an adverse event may berepresented by on a color scale ranging from yellow to red, or othersuitable color scales.

In some embodiments, the location and/or size of the marker may beassociated with a criticality level. The criticality level may representthe relative importance of an event, action, technique, phase or otheroccurrence identified by the marker. Accordingly, as used herein, theterm “criticality level” refers to any measure of an immediate need foran action to prevent hazardous result within a surgical procedure. Forexample, criticality level may include a numerical measure (such as“1.12”, “3.84”, “7”, “−4.01”, etc.), for example within a particularrange of values. In another example, criticality level may includefinite number of discrete levels (such as “Level 0”, “Level 1”, “Level2”, “High Criticality”, “Low Criticality”, “Non Critical”, etc.).

While color is provided as one example for distinguishing markerappearance to represent information, various other techniques may beused. For example, markers may have varying sizes, shapes, positions,orientations, font size, font types, font colors, marker animations, orother visual properties. In some embodiments, markers may be associatedwith different icons depending on the type of event, action, or phasewith which they are associated. For example, as shown in FIG. 4 , marker432, which may be associated with a decision junction, may have adifferent icon than marker 434, which may be associated with anothertype of event, such as a complication. The icon may represent the typeof intraoperative event associated with that location. For example,marker 436 may indicate that an incision occurs at this location in thevideo. The icons (or other visual properties) may be used to distinguishbetween unplanned events and planned events, types of errors (e.g.,miscommunication errors, judgment errors, or other forms of errors),specific adverse events that occurred, types of techniques beingperformed, the surgical phase being performed, locations ofintraoperative surgical events (e.g., in the abdominal wall, etc.), asurgeon performing the procedure, an outcome of the surgical procedure,or various other information.

In some exemplary embodiments, the one or more markers may include adecision making junction marker corresponding to a decision makingjunction of the surgical procedure. In some embodiments, such decisionmaking junction markers maybe visually distinct from other forms ortypes of markers. As an illustrative example, the decision makingjunction marker may have an icon indicating the location is associatedwith a decision making junction, as shown in FIG. 4 by marker 432. Asused herein, a decision making junction may refer to any part of aprocedure in which a decision is made, or in which a decision of aselected type of decisions or of a plurality of selected types ofdecisions is made. For example, the decision making junction marker mayindicate a location of a video depicting a surgical procedure wheremultiple courses of action are possible, and a surgeon opts to followone course over another. For example, the surgeon may decide whether todepart from a planned surgical procedure, to take a preventative action,to remove an organ or tissue, to use a particular instrument, to use aparticular surgical technique, or any other intraoperative decisions asurgeon may encounter. In one example, a decision making junction mayrefer to a part of a procedure in which a decision that has significanteffect on an outcome of the procedure is made. In another example,decision making junction may refer to a part of a procedure in which adecision that has no clear decision making guidelines has to be made. Inyet another example, a decision making junction may refer to a part of aprocedure in which a surgeon is faced with two or more viablealternatives, and where choosing the better alternative of the two ormore viable alternatives (for example, the alternative that is predictedto reduce a particular risk, the alternative that is predicted toimprove outcome, the alternative that is predicted to reduce cost, etc.)is based on at least a particular number of factors (for example, isbased on at least two factors, on at least five factors, on at least tenfactors, on at least one hundred factors, and so forth). In anadditional example, decision making junction may refer to a part of aprocedure in which a surgeon is faced with a decision of a particulartype, and where the particular type is included in a group of selecteddecision types.

The decision making junction may be detected using the computer analysisdescribed above. In some embodiments, video footage may be analyzed toidentify particular actions or sequences of actions performed by asurgeon that may indicate a decision has been made. For example, if thesurgeon pauses during a procedure, begins to use a different medicaldevice, or changes to a different course of action, this may indicate adecision has been made. In some embodiments, the decision makingjunction may be identified based on a surgical phase or intraoperativeevent identified in the video footage at that location. For example, anadverse event, such as a bleed, may be detected which may indicate adecision must be made on how to address the adverse event. As anotherexample, a particular phase of a surgical procedure may be associatedwith multiple possible courses of action. Accordingly, detecting thissurgical phase in the video footage may indicate a decision makingjunction. In some embodiments, a trained machine learning model may beused to identify the decision making junction. For example, a machinelearning model may be trained using training examples to detect decisionmaking junctions in videos, and the trained machine learning model maybe used to analyze the video and detect the decision making junction. Anexample of such training example may include a video clip, together witha label indicating locations of decision making junctions within thevideo clip, or together with a label indicating an absent of decisionmaking junctions in the video clip.

The selection of the decision making junction marker may enable thesurgeon to view two or more alternative video dips from two or morecorresponding other surgical procedures, thereby enabling the viewer tocompare alternative approaches. Alternative video clips may be any videoclips illustrating a procedure other than one currently being displayedto the user. Such an alternative may be drawn from other video footagenot included in the current video being output for display.Alternatively, if the current video footage includes a compilation ofdiffering procedures, the alternative footage may be drawn from adiffering location of the current video footage being displayed. Theother surgical procedures may be any surgical procedure other than thespecific procedure depicted in the current video being output fordisplay. In some embodiments, the other surgical procedures may be thesame type of surgical procedure depicted in the video being output fordisplay, but performed at different times, on different patients, and/orby different surgeons. In some embodiments, the other surgicalprocedures may not be the same type of procedure but may share the sameor similar decision making junctions as the one identified by thedecision making junction marker. In some embodiments, the two or morevideo dips may present differing conduct. For example, the two or morevideo clips may represent an alternate choice of action than the onetaken in the current video, as represented by the decision makingjunction marker.

The alternative video dips may be presented in various ways. In someembodiments, selecting the decision making junction marker mayautomatically cause display of the two or more alternative video clips.For example, one or more of the alternative video clips may be displayedin video playback region 410. In some embodiments, the video playbackregion may be split or divided to show one or more of the alternativevideo clips and/or the current video. In some embodiments, thealternative video clips may be displayed in another region, such asabove, below, or to the side of video playback region 410. In someembodiments, the alternative video clips may be displayed in a secondwindow, on another screen, or in any other space other than playbackregion 410. According to other embodiments, selecting the decisionmarker may open a menu or otherwise display options for viewing thealternative video dips. For example, selecting the decision namingmarker may pop up an alternative video menu containing depictions of theconduct in the associated alternative video clips. The alternative videoclips may be presented as thumbnails, text-based descriptions, videopreviews (e.g., playing a smaller resolution version or shortened dip),or the like. The menu may be overlaid on the video, may be displayed inconjunction with the video, or may be displayed in a separate area.

In accordance with embodiments of the present disclosure, the selectionof the decision making junction marker may cause a display of one ormore alternative possible decisions related to the selected decisionmaking junction marker. Similar to the alternative videos, thealternative possible decisions may be overlaid on the timeline and/orvideo, or may be displayed in a separate region, such as above, belowand/or to the side of the video, in a separate window, on a separatescreen, or in any other suitable manner. The alternative possibledecisions may be a list of alternative decisions the surgeon could havemade at the decision making junction. The list may also include images(e.g., depicting alternative actions), flow diagrams, statistics (e.g.,success rates, failure rates, usage rates, or other statisticalinformation), detailed descriptions, hyperlinks, or other informationassociated with the alternative possible decisions that may be relevantto the surgeon viewing the playback. Such a list may be interactive,enabling the viewer to select an alternative course of action from thelist and thereby cause video footage of the alternative course of actionto be displayed.

Further, in some embodiments, one or more estimated outcomes associatedwith the one or more alternative possible decisions may be displayed inconjunction with the display of the one or more alternative possibledecisions. For example, the list of alternative possible decisions mayinclude estimated outcomes of each of the alternative possibledecisions. The estimated outcomes may include an outcome that ispredicted to occur were the surgeon to have taken the alternativepossible decision. Such information may be helpful for trainingpurposes. For example, the surgeon may be able to determine that a moreappropriate action could have been taken than the one in the video andmay plan future procedures accordingly. In some embodiments, each of thealternative possible decisions may be associated with multiple estimatedoutcomes and a probability of each may be provided. The one or moreestimated outcomes may be determined in various ways. In someembodiments, the estimated outcomes may be based on known probabilitiesassociated with the alternative possible decisions. For example,aggregated data from previous surgical procedures with similar decisionmaking junctions may be used to predict the outcome of the alternativepossible decisions associated with the marker. In some embodiments, theprobabilities and/or data may be tailored to one or more characteristicsor properties of the current surgical procedure. For example, patientcharacteristics (such as a patient's medical condition, age, weight,medical history, or other characteristics), surgeon skill level,difficulty of the procedure, type of procedure, or other factors may beconsidered in determining the estimated outcomes. Other characteristicsmay also be analyzed, including the event characteristics describedabove with respect to video indexing.

In accordance with the present disclosure, the decision making junctionof the surgical procedure may be associated with a first patient, andthe respective similar decision making junctions may be selected frompast surgical procedures associated with patients with similarcharacteristics to the first patient. The past surgical procedures maybe preselected or automatically selected based on similar estimatedoutcomes as the respective similar decision making junctions, or becauseof similarities between the patient in the current video with thepatient's in the past surgical procedures. These similarities orcharacteristics may include a patient's gender, age, weight, height,physical fitness, heart rate, blood pressure, temperature, whether thepatient exhibits a particular medical condition or disease, medicaltreatment history, or any other traits or conditions that may berelevant.

Similarly, in some embodiments, the decision making junction of thesurgical procedure may be associated with a first medical professional,and the respective similar past decision making junctions may beselected from past surgical procedures associated with medicalprofessionals with similar characteristics to the first medicalprofessional. These characteristics may include, but are not limited to,the medical professional's age, medical background, experience level(e.g., the number of times the surgeon has performed this or similarsurgical procedures, the total number of surgical procedures the surgeonhas performed, etc.), skill level, training history, success rate forthis or other surgical procedures, or other characteristics that may berelevant.

In some exemplary embodiments, the decision making junction of thesurgical procedure is associated with a first prior event in thesurgical procedure, and the similar past decision making junctions areselected from past surgical procedures including prior events similar tothe first prior event. In one example, prior events may be determined tobe similar to the first prior event based on, for example, the type ofthe prior events, characteristics of the prior events, and so forth. Forexample, a prior event may be determined as similar to the first priorevent when a similarity measure between the two is above a selectedthreshold. Some non-limiting examples of such similarity measures aredescribed above. The occurrence and/or characteristics of the priorevent may be relevant for determining estimated outcomes for thealternative possible decisions. For example, if the surgeon runs intocomplications with a patient, the complications may at least partiallybe determinative of the most appropriate outcome, whereas a differentoutcome may be appropriate in absence of the complications. The firstprior event may include, but is not limited to, any of theintraoperative events described in detail above. Some non-limitingcharacteristics of the first prior may include any of the eventcharacteristics described above. For example, the first prior event mayinclude an adverse event or complication, such as bleeding, mesentericemphysema, injury, conversion to unplanned open, incision significantlylarger than planned, hypertension, hypotension, bradycardia, hypoxemia,adhesions, hernias, atypical anatomy, dural tears, periorator injury,arterial occlusions, and so forth. The first prior event may alsoinclude positive or planned events, such as a successful incision,administration of a drug, usage of a surgical instrument, an excision, aresection, a ligation, a graft, suturing, stitching, or any other event.

In accordance with the present disclosure, the decision making junctionof the surgical procedure may be associated with a medical condition,and the respective similar decision making junctions may be selectedfrom past surgical procedures associated with patients with similarmedical conditions. The medical conditions may include any condition ofthe patient related to the patient's health or well-being. In someembodiments, the medical condition may be the condition being treated bythe surgical procedure. In other embodiments, the medical condition maybe a separate medical condition. The medical condition may be determinedin various ways. In some embodiments, the medical condition may bedetermined based on data associated with the plurality of videos. Forexample, the video may be tagged with information including the medicalcondition. In other embodiments, the medical condition may be determinedby an analysis of the at least one video and may be based on anappearance of an anatomical structure appearing in the at least onevideo. For example, the color of a tissue, the relative color of onetissue with respect to the color of another tissue, size of an organ,relative size of one organ with respect to a size of another organ,appearance of a gallbladder or other organ, presence of lacerations orother marks, or any other visual indicators associated with ananatomical structure, may be analyzed to determine the medicalcondition. In one example, a machine learning model may be trained usingtraining examples to determine medical conditions from videos, and thetrained machine learning model may be used to analyze the at least onevideo footage and determine the medical condition. An example of suchtraining example may include a video clip of a surgical procedure,together with a label indicating one or more medical conditions.

In some aspects of the present disclosure, information related to adistribution of past decisions made in respective similar past decisionmaking junctions may be displayed in conjunction with the display of thealternative possible decisions. For example, as described above, aparticular decision making junction may be associated with multiplepossible decisions for a course of action. The past decisions mayinclude decisions that were made by surgeons in previous surgicalprocedures when faced with the same or similar decision making junction.For example, each of the past decisions may correspond to one of thealternate possible decisions described above. Accordingly, as usedherein, respective similar past decision making junctions refers to thedecision making junction that occurred in the past surgical procedurewhen the past decision was made. In some embodiments, the respectivesimilar past decision making junctions may be the same as the decisionmaking junction identified by the marker. For example, if the decisionmaking junction is an adverse event, such as a bleed, the past decisionsmay correspond to how other surgeons have addressed the bleed inprevious surgical procedures. In other embodiments, the decision makingjunction may not be identical, but may be similar. For example, thepossible decisions made by surgeons encountering a dural tear may besimilar to other forms of tears and, accordingly, a distribution of pastdecisions associated with a dural tear may be relevant to the otherforms of tears. The past decisions may be identified by analyzing videofootage, for example, using the computer analysis techniques describedabove. In some embodiments, the past decisions may be indexed using thevideo indexing techniques described above, such that they can be readilyaccessed for displaying a distribution of past decisions. In oneexample, the distribution may include a conditional distribution, forexample presenting a distribution of past decisions made in respectivesimilar past decision making junctions that has a common property. Inanother example, the distribution may include an unconditionaldistribution, for example presenting a distribution of past decisionsmade in all respective similar past decision making junctions.

The displayed distribution may indicate how common each of the possibledecisions were among the other alternative possible decisions associatedwith the respective similar past decision making junctions. In someembodiments, the displayed distribution may include a number of timeseach of the decisions was made. For example, a particular decisionmaking junction may have three alternative possible decisions: decisionA, decision B, and decision C. Based on the past decisions made insimilar decision making junctions, the number of times each of thesealternative possible decisions has been performed may be determined. Forexample, decision A may have been performed 167 times, decision B mayhave been performed 47 times, and decision C may have been performed 13times. The distribution may be displayed as a list of each of thealternative possible decisions, along with the number of times they havebeen performed. The displayed distribution may also indicate therelative frequency of each of the decisions, for example, by displayingratios, percentages, or other statistical information. For example, thedistribution may indicate that decisions A. B and C have been performedin 73.6%, 20.7% and 5.7% of past decisions, respectively. In someembodiments, the distribution may be displayed as a graphicalrepresentation of the distribution, such as a bar graph, a histogram, apie chart, a distribution curve, or any other graphical representationthat may be used to show distribution.

In some embodiments, only a subset of the decisions may be displayed.For example, only the most common decisions may be displayed based onthe number of times the decision was made (e.g., exceeding a thresholdnumber of times, etc.). Various methods described above for identifyingthe similar past decision making junctions may be used, includingidentifying surgical procedures associated with similar medicalconditions, patient characteristics, medical professionalcharacteristics, and/or prior events.

In some embodiments, the one or more estimated outcomes may be a resultof an analysis of a plurality of videos of past surgical proceduresincluding respective similar decision making junctions. For example, arepository of video footage may be analyzed using various computeranalysis techniques, such as the object and/or motion detectionalgorithms described above, to identify videos including decision makingjunctions that are the same as or share similar characteristics with thedecision making junction identified by the marker. This may includeidentifying other video footage having the same or similar surgicalphases, intraoperative surgical events, and/or event characteristics asthose that were used to identify the decision making junction in thevideo presented in the timeline. The outcomes of the alternativepossible decisions may be estimated based on the outcomes in the pastsurgical procedures. For example, if a particular method of performing asuture consistently results in a full recovery by the patient, thisoutcome may be estimated for this possible decision and may be displayedon the timeline.

In some exemplary embodiments, the analysis may include usage of animplementation of a computer vision algorithm. The computer visionalgorithm may be the same as or similar to any of the computer visionalgorithms described above. One example of such computer algorithm mayinclude the object detection and tracking algorithms described above.Another example of such computer vision algorithm may include usage of atrained machine learning model. Other non-limiting examples of suchcomputer vision algorithm are described above. For example, if thedecision making junction marker was identified based on a particularadverse event occurring in the video, other video footage having thesame or similar adverse events may be identified. The video footage mayfurther be analyzed to determine an outcome of the decision made in pastsurgical video. This may include the same or similar computer analysistechniques described above. In some embodiments, this may includeanalyzing the video to identify the result of the decision. For example,if the decision making junction is associated with an adverse eventassociated with an anatomical structure, such as a tear, the anatomicalstructure may be assessed at various frames after the decision todetermine whether the adverse event was remediated, how quickly it wasremediated, whether additional adverse events occurred, whether thepatient survived, or other indicators of the outcome.

In some embodiments, additional information may also be used todetermine the outcome. For example, the analysis may be based on one ormore electronic medical records associated with the plurality of videosof past surgical procedures. For example, determining the outcome mayinclude referencing an electronic medical record associated with thevideo in which a particular decision was made to determine whether thepatient recovered, how quickly the patient recovered, whether there wereadditional complications, or the like. Such information may be useful inpredicting the outcome that may result at a later time, outside of thescope of the video footage. For example, the outcome may be severaldays, weeks, or months after the surgical procedure. In someembodiments, the additional information may be used to inform theanalysis of which videos to include in the analysis. For example, usinginformation gleaned from the medical records, videos sharing similarpatient medical history, disease type, diagnosis type, treatment history(including past surgical procedures), healthcare professionalidentities, healthcare professional skill levels, or any other relevantdata may be identified. Videos sharing these or other characteristicsmay provide a more accurate idea of what outcome can be expected foreach alternative possible decision.

The similar decision making junctions may be identified based on howclosely they correlate to the current decision making junction. In someembodiments, the respective similar decision making junctions may besimilar to the decision making junction of the surgical procedureaccording to a similarity metric. The metric may be any value,classification, or other indicator of how closely the decision makingjunctions are related. Such a metric may be determined based on thecomputer vision analysis in order to determine how closely theprocedures or techniques match. The metric may also be determined basedon the number of characteristics the decision making junctions have incommon and the degree to which the characteristics match. For example,two decision making junctions with patients having similar medicalconditions and physical characteristics may be assigned a highersimilarity based on the similarity metric than two more distinctivepatients. Various other characteristics and/or considerations may alsobe used. Additionally or alternatively, the similarity metric may bebased on any similarity measure, such as the similarity measuresdescribed above. For example, the similarity metric may be identical tothe similarity measure, may be a function of the similarity measure, andso forth.

Various other marker types may be used in addition to or instead ofdecision making junction markers. In some embodiments, the markers mayinclude intraoperative surgical event markers, which may be associatedwith locations in the video associated with the occurrence of aninteroperative event. Examples of various intraoperative surgical eventsthat may be identified by the markers are provided throughout thepresent disclosure, including in relation to the video indexingdescribed above. In some embodiments, the intraoperative surgical eventmarkers may be generic markers, indicating that an intraoperativesurgical event occurred at that location. In other embodiments, theintraoperative surgical event markers may identify a property of theintraoperative surgical event, including the type of the event, whetherthe event was an adverse event, or any other characteristic. Examplemarkers are shown in FIG. 4 . As an illustrative example, the icon shownfor marker 434 may be used to represent a generic intraoperativesurgical event marker. Marker 436 on the other hand, may represent amore specific intraoperative surgical event marker, such as identifyingthat an incision occurred at that location. The markers shown in FIG. 4are provided by way of example, and various other forms of markers maybe used.

These intraoperative surgical event markers may be identifiedautomatically, as described above. Using the computer analysis methodsdescribed above, medical instruments, anatomical structures, surgeoncharacteristics, patient characteristics, event characteristics, orother features may be identified in the video footage. For example, theinteraction between an identified medical instrument and an anatomicalstructure may indicate that an incision, a suturing, or otherintraoperative event is being performed. In some embodiments, theintraoperative surgical event markers may be identified based oninformation provided in a data structure, such as data structure 600described above in reference to FIG. 6 .

Consistent with the disclosed embodiments, selection of anintraoperative surgical event marker may enable the surgeon to viewalternative video clips from differing surgical procedures. In someembodiments, the alternative video clips may present differing ways inwhich a selected intraoperative surgical event was handled. For example,in the current video the surgeon may perform an incision or other actionaccording to one technique. Selecting the intraoperative surgical eventmarkers may allow the surgeon to view alternative techniques that may beused to perform the incision or other action. In another example, theintraoperative surgical event may be an adverse event, such as a bleed,and the alternative video clips may depict other ways surgeons havehandled the adverse event. In some embodiments, where the markers relateto intraoperative surgical events, the selection of an intraoperativesurgical event marker may enable the surgeon to view alternative videoclips from differing surgical procedures. For example, the differingsurgical procedures may be of a different type (such as a laparoscopicsurgery versus thoracoscopic surgery) but may still include the same orsimilar intraoperative surgical events. The surgical procedures may alsodiffer in other ways, including differing medical conditions, differingpatient characteristics, differing medical professionals, or otherdistinctions. Selecting the intraoperative surgical event marker mayallow the surgeon to view alternative video clips from the differingsurgical procedures.

The alternative video clips may be displayed in various ways, similar toother embodiments described herein. For example, selecting theintraoperative surgical event markers may cause a menu to be displayed,from which the surgeon may select the alternative video clips. The menumay include descriptions of the differing ways in which the selectedintraoperative surgical event was handled, thumbnails of the videoclips, previews of the video clips, and/or other information associatedwith the video clips, such as the dates they were recorded, the type ofsurgical procedure, a name or identity of a surgeon performing thesurgical procedure, or any other relevant information.

In accordance with some embodiments of the present disclosure, the atleast one video may include a compilation of footage from a plurality ofsurgical procedures, arranged in procedural chronological order.Procedural chronological order may refer to the order events occurrelative to a surgical procedure. Accordingly, arranging a compilationof footage in procedural chronological order may include arranging thedifferent events from differing patients in the order in which theywould have occurred if the procedure had been conducted on a singlepatient. In other words, although compiled from various surgeries ondiffering patients, playback of the compilation will display the footagein the order the footage would appear within the surgical procedure. Insome embodiments, the compilation of footage may depict complicationsfrom the plurality of surgical procedures. In such embodiments, the oneor more markers may be associated with the plurality of surgicalprocedures and may be displayed on a common timeline. Thus, although aviewer interacts with a single timeline, the video footage presentedalong the timeline may be derived from differing procedures and/ordiffering patients. Example complications that may be displayed aredescribed above with respect to video indexing.

FIG. 5 is a flowchart illustrating an example process 500 for reviewingsurgical videos, consistent with the disclosed embodiments. Process 500may be performed by at least one processor, such as one or moremicroprocessors. In some embodiments, process 500 is not necessarilylimited to the steps illustrated, and any of the various embodimentsdescribed herein may also be included in process 500. At step 510,process 500 may include accessing at least one video of a surgicalprocedure, for example as described above. The at least one video mayinclude video footage from a single surgical procedure or may be acompilation of footage from a plurality of procedures, as previouslydiscussed. Process 500 may include causing the at least one video to beoutput for display in step 520. As described above, causing the at leastone video to be output for display may include sending a signal forcausing display of the at least one video on a screen or other displaydevice, storing the at least one video in a location accessible toanother computing device, transmitting the at least one video, or anyother process or steps that may cause the video to be displayed.

At step 530, process 500 may include overlaying on the at least onevideo outputted for display a surgical timeline, wherein the surgicaltimeline includes markers identifying at least one of a surgical phase,an intraoperative surgical event, and a decision making junction. Insome embodiments, the surgical timeline may be represented as ahorizontal bar displayed along with the video. The markers may berepresented as shapes, icons, or other graphical representations alongthe timeline. FIG. 4 provides an example of such an embodiment. In otherembodiments, the timeline may be a text-based list of phases, events,and/or decision making junctions in chronological order. The markers maysimilarly be text-based and may be included in the list.

Step 540 may include enabling a surgeon, while viewing playback of theat least one video, to select one or more markers on the surgicaltimeline, and thereby cause a display of the video to skip to a locationassociated with the selected marker. In some embodiments, the surgeonmay be able to view additional information about the event or occurrenceassociated with the marker, which may include information from pastsurgical procedures. For example, the markers may be associated with anintraoperative surgical event and selecting the marker may enable thesurgeon to view alternative video clips of past surgical proceduresassociated with the intraoperative surgical event. For example, thesurgeon may be enabled to view clips from other surgeries where asimilar intraoperative surgical event was handled differently, where adifferent technique was used, or where an outcome varied. In someembodiments, the marker may be a decision making junction marker,representing a decision that was made during the surgical procedure.Selecting the decision making junction marker may enable the surgeon toview information about the decision, including alternative decisions.Such information may include videos of past surgical proceduresincluding similar decision making junctions, a list or distribution ofalternate possible decisions, estimated outcomes of the alternatepossible decisions, or any other relevant information. Based on thesteps described in process 500, the surgeon or other users may be ableto more effectively and more efficiently review surgical videos usingthe timeline interface.

In preparing for a surgical procedure, it is often beneficial forsurgeons to review videos of similar surgical procedures that have beenperformed. It may be too cumbersome and time consuming, however, for asurgeon to identify relevant videos or portions of videos in preparingfor a surgical procedure. Therefore, there is a need for unconventionalapproaches that efficiently, effectively index surgical video footagebased on contents of the footage such that it may be easily accessed andreviewed by a surgeon or other medical professional.

Aspects of this disclosure may relate to video indexing, includingmethods, systems, devices, and computer readable media. For example,surgical events within surgical phases may be automatically detected insurgical footage. Viewers may be enabled to skip directly to an event,to view only events with specified characteristics, and so forth. Insome embodiments, a user may specify within a surgical phase (e.g., adissection) an event (e.g., inadvertent injury to an organ) having acharacteristic (e.g., a particular complication), so that the user maybe presented with video dips of one or more events sharing thatcharacteristic.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In a broadest sense, the method is not limited toparticular physical and/or electronic instrumentalities, but rather maybe accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve accessingvideo footage to be indexed, the video footage to be indexed includingfootage of a particular surgical procedure. As used herein, video mayinclude any form of recorded visual media including recorded imagesand/or sound. For example, a video may include a sequence of one or moreimages captured by an image capture device, such as cameras 115, 121,123, and/or 125, as described above in connection with FIG. 1 . Theimages may be stored as individual files or may be stored in a combinedformat, such as a video file, which may include corresponding audiodata. In some embodiments, video may be stored as raw data and/or imagesoutput from an image capture device. In other embodiments the video maybe processed. For example, video files may include Audio VideoInterleave (AVI), Flash Video Format (FLV), QuickTime File Format (MOV),MPEG (MPG, MP4, M4P, etc.), Windows Media Video (WMV), Material ExchangeFormat (MXF), uncompressed format, lossy compressed format, losslesscompressed format, or any other suitable video file formats.

Video footage may refer to a length of video that has been captured byan image capture device. In some embodiments, video footage may refer toa length of video that includes a sequence of images in the order theywere originally captured in. For example, video footage may includevideo that has not been edited to form a video compilation. In otherembodiments, video footage may be edited in one or more ways, such as toremove frames associated with inactivity, or to otherwise compile framesnot originally captured sequentially. Accessing the video footage mayinclude retrieving video footage from a storage location, such as amemory device. The video footage may be accessed from a local memory,such as a local hard drive, or may be accessed from a remote source, forexample, through a network connection. Consistent with the presentdisclosure, indexing may refer to a process for storing data such thatit may be retrieved more efficiently and/or effectively. Indexing videofootage may include associating one or more properties or indicatorswith the video footage such that the video footage may be identifiedbased on the properties or indicators.

A surgical procedure may include any medical procedure associated withor involving manual or operative procedures on a patient's body.Surgical procedures may include cutting, abrading, suturing, or othertechniques that involve physically changing body tissues and organs.Some examples of such surgical procedures may include a laparoscopicsurgery, a thoracoscopic procedure, a bronchoscopic procedure, amicroscopic procedure, an open surgery, a robotic surgery, anappendectomy, a carotid endarterectomy, a carpal tunnel release, acataract surgery, a cesarean section, a cholecystectomy, a colectomy(such as a partial colectomy, a total colectomy, etc.), a coronaryangioplasty, a coronary artery bypass, a debridement (for example of awound, a burn, an infection, etc.), a free skin graft, ahemorrhoidectomy, a hip replacement, a hysterectomy, a hysteroscopy, aninguinal hernia repair, a knee arthroscopy, a knee replacement, amastectomy (such as a partial mastectomy, a total mastectomy, a modifiedradical mastectomy, etc.), a prostate resection, a prostate removal, ashoulder arthroscopy, a spine surgery (such as a spinal fusion, alaminectomy, a foraminotomy, a discectomy, a disk replacement, aninterlaminar implant, etc.), a tonsillectomy, a cochlear implantprocedure, brain tumor (for example meningioma, etc.) resection,interventional procedures such as percutaneous transluminal coronaryangioplasty, transcatheter aortic valve replacement, minimally Invasivesurgery for intracerebral hemorrhage evacuation, or any other medicalprocedure involving some form of incision. While the present disclosureis described in reference to surgical procedures, it is to be understoodthat it may also apply to other forms of medical procedures, orprocedures generally.

In some exemplary embodiments, the accessed video footage may includevideo footage captured via at least one image sensor located in at leastone of a position above an operating table, in a surgical cavity of apatient, within an organ of a patient or within vasculature of apatient. An image sensor may be any sensor capable of recording video.An image sensor located in a position above an operating table mayinclude any image sensor placed external to a patient configured tocapture images from above the patient. For example, the image sensor mayinclude cameras 115 and/or 121, as shown in FIG. 1 . In otherembodiments, the image sensor may be placed internal to the patient,such as, for example, in a cavity. As used herein, a cavity may includeany relatively empty space within an object. Accordingly, a surgicalcavity may refer to a space within the body of a patient where asurgical procedure or operation is being performed, or where surgicaltools are present and/or used. It is understood that the surgical cavitymay not be completely empty but may include tissue, organs, blood orother fluids present within the body. An organ may refer to anyself-contained region or part of an organism. Some examples of organs ina human patient may include a heart or liver. A vasculature may refer toa system or grouping of blood vessels within an organism. An imagesensor located in a surgical cavity, an organ, and/or a vasculature mayinclude a camera included on a surgical tool inserted into the patient.

Aspects of this disclosure may include analyzing the video footage toidentify a video footage location associated with a surgical phase ofthe particular surgical procedure. As used herein with respect to videofootages, a location may refer any particular position or range withinthe video footage. In some embodiments the location may include aparticular frame or range of frames of a video. Accordingly, videofootage locations may be represented as one or more frame numbers orother identifiers of a video footage file. In other embodiments, thelocation may refer to a particular time associated with the videofootage. For example, a video footage location may refer to a time indexor timestamp, a time range, a particular starting time and/or endingtime, or any other indicator of position within the video footage. Inother embodiments, the location may refer to at least one particularposition within at least one frame. Accordingly, video footage locationsmay be represented as one or more pixels, voxels, bounding boxes,bounding polygons, bounding shapes, coordinates, and so forth.

For the purposes of the present disclosure, a phase may refer to aparticular period or stage of a process or series of events.Accordingly, a surgical phase may refer to a particular period or stageof a surgical procedure, as described above. For example, surgicalphases of a laparoscopic cholecystectomy surgery may include trocarplacement, preparation, calot's triangle dissection, clipping andcutting of cystic duct and artery, gallbladder dissection, gallbladderpackaging, cleaning and coagulation of liver bed, gallbladderretraction, and so forth. In another example, surgical phases of acataract surgery may include preparation, povidone-iodine injection,corneal incision, capsulorhexis, phaco-emulsification, corticalaspiration, intraocular lens implantation, intraocular-lens adjustment,wound sealing, and so forth. In yet another example, surgical phases ofa pituitary surgery may include preparation, nasal incision, noseretractor installation, access to the tumor, tumor removal, column ofnose replacement, suturing, nose compress installation, and so forth.Some other examples of surgical phases may include preparation,incision, laparoscope positioning, suturing, and so forth.

In some embodiments, identifying the video footage location may be basedon user input. User input may include any information provided by auser. As used with respect to video indexing, the user input may includeinformation relevant to identifying the video footage location. Forexample, a user may input a particular frame number, timestamp, range oftimes, start times and/or stop times, or any other information that mayidentify a video footage location. Alternatively, the user input mightinclude entry or selection of a phase, event, procedure, or device used,which input may be associated with particular video footage (e.g., forexample through a lookup table or other data structure). The user inputmay be received through a user interface of a user device, such as adesktop computer, a laptop, a table, a mobile phone, a wearable device,an internet of things (IoT) device, or any other means for receivinginput from a user. The interface may include, for example, one or moredrop down menus with one or more pick lists of phase names; a data entryfield that permits the user to enter the phase name and/or that suggestsphase names once a few letters are entered; a pick list from which phasenames may be chosen; a group of selectable icons each associated with adiffering phase, or any other mechanism that allows users to identify orselect a phase. For example, a user may input the phase name through auser interface similar to user interface 700, as described in greaterdetail below with respect to FIG. 7 . In another example, the user inputmay be received through voice commands and/or voice inputs, and the userinput may be processed using speech recognition algorithms. In yetanother example, the user input may be received through gestures (suchas hand gestures), and the user input may be processed using gesturerecognition algorithms.

In some embodiments, identifying the video footage location may includeusing computer analysis to analyze frames of the video footage. Computeranalysis may include any form of electronic analysis using a computingdevice. In some embodiments, computer analysis may include using one ormore image recognition algorithms to identify features of one or moreframes of the video footage. Computer analysis may be performed onindividual frames, or may be performed across multiple frames, forexample, to detect motion or other changes between frames. In someembodiments computer analysis may include object detection algorithms,such as Viola-Jones object detection, scale-invariant feature transform(SIFT), histogram of oriented gradients (HOG) features, convolutionalneural networks (CNN), or any other forms of object detectionalgorithms. Other example algorithms may include video trackingalgorithms, motion detection algorithms, feature detection algorithms,color-based detection algorithms, texture based detection algorithms,shape based detection algorithms, boosting based detection algorithms,face detection algorithms, or any other suitable algorithm for analyzingvideo frames. In one example, a machine learning model may be trainedusing training examples to identify particular locations within videos,and the trained machine learning model may be used to analyze the videofootage and identify the video footage location. An example of suchtraining example may include a video clip together with a labelindicating a location within a video clip, or together with a labelindicating that no corresponding location is included within the videoclip.

In some embodiments, the computer image analysis may include using aneural network model trained using example video frames includingpreviously-identified surgical phases to thereby identify at least oneof a video footage location or a phase tag. In other words, frames ofone or more videos that are known to be associated with a particularsurgical phase may be used to train a neural network model, for exampleusing a machine learning algorithm, using back propagation, usinggradient descent optimization, and so forth. The trained neural networkmodel may therefore be used to identify whether one or more video framesare also associated with the surgical phase. Some non-limiting examplesof such artificial neural networks may comprise shallow artificialneural networks, deep artificial neural networks, feedback artificialneural networks, feed forward artificial neural networks, autoencoderartificial neural networks, probabilistic artificial neural networks,time delay artificial neural networks, convolutional artificial neuralnetworks, recurrent artificial neural networks, long short term memoryartificial neural networks, and so forth. In some embodiments, thedisclosed methods may further include updating the trained neuralnetwork model based on at least one of the analyzed frames.

In some aspects of the present disclosure, analyzing the video footageto identify the video footage location associated with at least one ofthe surgical event or the surgical phase may include performing computerimage analysis on the video footage to identify at least one of abeginning location of the surgical phase for playback or a beginning ofa surgical event for playback. In other words, using the computeranalysis techniques discussed above, the disclosed methods may includeidentifying a location within the video footage where a surgical phaseor event begins. For example, the beginning of a surgical event, such asan incision, may be detected using the object and/or motion detectionalgorithms described above. In other embodiments, the beginning of theincision may be detected based on machine learning techniques. Forexample, a machine learning model may be trained using video footage andcorresponding label indicating known beginning points of an incision orother surgical events and/or procedures. The trained model may be usedto identify similar procedure and/or event beginning locations withinother surgical video footage.

Some aspects of this disclosure may include generating a phase tagassociated with the surgical phase. As used herein, a “tag” may refer toany process or marker by which information is associated with or linkedto a set of data. In some embodiments, a tag may be a property of a datafile, such as a video file. Accordingly, generating the tag may includewriting or overwriting properties within a video file. In someembodiments, generating a tag may include writing information to a fileother than the video file itself, for example, by associating the videofile with the tag in a separate database. The tag may be expressed astextual information, a numerical identifier, or any other suitable meansfor tagging. A phase tag may be a tag that identifies a phase of asurgical phase, as described above. In one embodiment, a phase tag maybe a marker indicating a location in video where a surgical phasebegins, a marker indicating a location in video where a surgical phaseends, a marker indicating a location in video in the middle of asurgical phase, or indicating a range of video encompassing the surgicalphase. The tag may be a pointer in the video data itself or may belocated in a data structure to permit a lookup of a phase location. Thephase tag may include computer readable information for causing displayof the phase and may also include human-readable information foridentifying the phase to a user. For example, generating a phase tagassociated with the surgical phase may include generating a tagincluding text such as “laparoscope positioning” to indicate the taggeddata is associated with that phase of the surgical procedure. In anotherexample, generating a phase tag associated with the surgical phase mayinclude generating a tag including binary encoding of a surgical phaseidentifier. In some embodiments, generating the phase tag may be basedon a computer analysis of video footage depicting the surgical phase.For example, the disclosed methods may include analyzing footage of thesurgical phase using the object and motion detection analysis methodsdescribed above to determine the phase tag. For example, if it is knownthat a phase begins or ends using a particular type of medical device orother instrumentality used in a unique way or in a unique order, imagerecognition may be performed on the video footage to identify aparticular phase through image recognition performed to identify theunique use of the instrumentality to identify a particular phase.Generating the phase tag may also include using a trained machinelearning model or a neural network model (such as deep neural network,convolutional neural networks, etc.), which may be trained to associateone or more video frames with one or more phase tags. For example,training examples may be fed to a machine learning algorithm to developa model configured to associate other video footage data with one ormore phase tags. An example of such training example may include a videofootage together with a label indicating the desired tags or the absentof desired tags corresponding to the video footage. Such label mayinclude an indication of one or more locations within the video footagecorresponding to the surgical phase, an indication of a type of thesurgical phase, an indication of properties of the surgical phase, andso forth.

A method in accordance with the present disclosure may includeassociating the phase tag with the video footage location. Any suitablemeans may be used to associate the phase tag with the video footagelocation. Such tag may include an indication of one or more locationswithin the video footage corresponding to the surgical phase, anindication of a type of the surgical phase, an indication of propertiesof the surgical phase, and so forth. In some embodiments, the videofootage location may be included in the tag. For example, the tag mayinclude a timestamp, time range, frame number, or other means forassociating the phase tag to the video footage location. In otherembodiments, the tag may be associated with the video footage locationin a database. For example, the database may include information linkingthe phase tag to the video footage and to the particular video footagelocation. The database may include a data structure, as described infurther detail below.

Embodiments of the present disclosure may further include analyzing thevideo footage to identify an event location of a particularintraoperative surgical event within the surgical phase. Anintraoperative surgical event may be any event or action that occursduring a surgical procedure or phase. In some embodiments, anintraoperative surgical event may include an action that is performed aspart of a surgical procedure, such as an action performed by a surgeon,a surgical technician, a nurse, a physician's assistant, ananesthesiologist, a doctor, or any other healthcare professional. Theintraoperative surgical event may be a planned event, such as anincision, administration of a drug, usage of a surgical instrument, anexcision, a resection, a ligation, a graft, suturing, stitching, or anyother planned event associated with a surgical procedure or phase. Insome embodiments, the intraoperative surgical event may include anadverse event or a complication. Some examples of intraoperative adverseevents may include bleeding, mesenteric emphysema, injury, conversion tounplanned open surgery (for example, abdominal wall incision), incisionsignificantly larger than planned, and so forth. Some examples ofintraoperative complications may include hypertension, hypotension,bradycardia, hypoxemia, adhesions, hernias, atypical anatomy, duraltears, periorator injury, arterial occlusions, and so forth. Theintraoperative event may include other errors, including technicalerrors, communication errors, management errors, judgment errors,decision making errors, errors related to medical equipment utilization,miscommunication, and so forth.

The event location may be a location or range within the video footageassociated with the intraoperative surgical event. Similar to the phaselocation described above, the event location may be expressed in termsof particular frames of the video footage (e.g., a frame number or arange of frame numbers) or based on time information (e.g., a timestamp,a time range, or beginning and end times), or any other means foridentifying a location within the video footage. In some embodiments,analyzing the video footage to identify the event location may includeusing computer analysis to analyze frames of the video footage. Thecomputer analysis may include any of the techniques or algorithmsdescribed above. As with phase identification, event identification maybe based on a detection of actions and instrumentalities used in a waythat uniquely identifies an event. For example, image recognition mayidentify when a particular organ is incised, to enable marking of thatincision event. In another example, image recognition may be used tonote the severance of a vessel or nerve, to enable marking of thatadverse event. Image recognition may also be used to mark events bydetection of bleeding or other fluid loss. In some embodiments,analyzing the video footage to identify the event location may includeusing a neural network model (such as a deep neural network, aconvolutional neural network, etc.) trained using example video framesincluding previously-identified surgical events to thereby identify theevent location. In one example, a machine learning model may be trainedusing training examples to identify locations of intraoperative surgicalevents in portions of videos, and the trained machine learning model maybe used to analyze the video footage (or a portion of the video footagecorresponding to the surgical phase) and identify the event location ofthe particular intraoperative surgical event within the surgical phase.An example of such training example may include a video clip togetherwith a label indicating a location of a particular event within thevideo clip, or an absence of such event.

Some aspects of the present disclosure may involve associating an eventtag with the event location of the particular intraoperative surgicalevent. As discussed above, a tag may include any means for associatinginformation with data or a portion of data. An event tag may be used toassociate data or portions of data with an event, such as anintraoperative surgical event. Similar to the phase tag, associating theevent tag with the event location may include writing data to a videofile, for example, to the properties of the video file. In otherembodiments, associating the event tag with the event location mayinclude writing data to a file or database associating the event tagwith the video footage and/or the event location. Alternatively,associating an event tag with an event location may include recording amarker in a data structure, where the data structure correlates a tagwith a particular location or range of locations in video footage. Insome embodiments, the same file or database may be used to associate thephase tag to the video footage as the event tag. In other embodiments, aseparate file or database may be used.

Consistent with the present disclosure, the disclosed methods mayinclude storing an event characteristic associated with the particularintraoperative surgical event. The event characteristic may be any traitor feature of the event. For example, the event characteristic mayinclude properties of the patient or surgeon, properties orcharacteristics of the surgical event or surgical phase, or variousother traits. Examples of features may include, excessive fatty tissue,an enlarged organ, tissue decay, a broken bone, a displaced disc, or anyother physical characteristic associated with the event. Somecharacteristics may be discernable by computer vision, and others may bediscernable by human input. In the latter example, the age or age rangeof a patient may be stored as an event characteristic. Similarly,aspects of a patient's prior medical history may be stored as an eventcharacteristic (e.g., patient with diabetes). In some embodiments, thestored event characteristic may be used to distinguish intraoperativesurgical events from other similar events. For example, a medicalpractitioner may be permitted to search video footage to identify one ormore coronary artery bypass surgeries performed on males over the age of70 with arrhythmia. Various other examples of stored eventcharacteristics that may be used are provided below.

The stored event characteristic may be determined in various ways. Someaspects of the disclosed methods may involve determining the storedevent characteristic based on user input. For example, a user may inputthe event characteristic to be stored via a user interface similar towhat was described above in connection with the selection of a phase oran event. In another example, a user may input the event characteristicto be stored via voice commands. Various examples of such uses areprovided below. Other aspects of the disclosed methods may involvedetermining the stored event characteristic based on a computer analysisof video footage depicting the particular intraoperative surgical event.For example, the disclosed methods may include using various imageand/or video analysis techniques as described above to recognize eventcharacteristics based on the video footage. As an illustrative example,the video footage may include a representation of one or more anatomicalstructures of a patient and an event characteristic identifying theanatomical structures may be determined based on detecting theanatomical structure in the video footage, or based on detecting theinteraction between a medical instrument and the anatomical structure.In another example, a machine learning model may be trained usingtraining examples to determine event characteristics from videos, andthe trained machine learning model may be used to analyze the videofootage and determine the stored event characteristic. An example ofsuch training example may include a video clip depicting anintraoperative surgical event together with a label indicating acharacteristic of the event.

Some aspects of the present disclosure may include associating at leasta portion of the video footage of the particular surgical procedure withthe phase tag, the event tag, and the event characteristic in a datastructure that contains additional video footage of other surgicalprocedures, wherein the data structure also includes respective phasetags, respective event tags, and respective event characteristicsassociated with one or more of the other surgical procedures. A datastructure consistent with this disclosure may include any collection ofdata values and relationships among them. The data may be storedlinearly, horizontally, hierarchically, relationally, non-relationally,uni-dimensionally, multidimensionally, operationally, in an orderedmanner, in an unordered manner, in an object-oriented manner, in acentralized manner, in a decentralized manner, in a distributed manner,in a custom manner, in a searchable repository, in a sorted repository,in an indexed repository, or in any manner enabling data access. By wayof non-limiting examples, data structures may include an array, anassociative array, a linked list, a binary tree, a balanced tree, aheap, a stack, a queue, a set, a hash table, a record, a tagged union,ER model, and a graph. For example, a data structure may include an XMLdatabase, an RDBMS database, an SQL database or NoSQL alternatives fordata storage/search such as, for example, MongoDB, Redis, Couchbase,Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra,Amazon DynamoDB, Scylla, HBase, and Neo4J. A data structure may be acomponent of the disclosed system or a remote computing component (e.g.,a cloud-based data structure). Data in the data structure may be storedin contiguous or non-contiguous memory. Moreover, a data structure, asused herein, does not require information to be co-located. It may bedistributed across multiple servers, for example, that may be owned oroperated by the same or different entities. Thus, for example, a datastructure may include any data format that may be used to associatevideo footage with phase tags, event tags, and/or event characteristics.

FIG. 6 illustrates an example data structure 600 consistent with thedisclosed embodiments. As shown in FIG. 6 , data structure 600 maycomprise a table including video footage 610 and video footage 620pertaining to different surgical procedures. For example, video footage610 may include footage of a laparoscopic cholecystectomy, while videofootage 620 may include footage of a cataract surgery. Video footage 620may be associated with footage location 621, which may correspond to aparticular surgical phase of the cataract surgery. Phase tag 622 mayidentify the phase (in this instance a corneal incision) associated withfootage location 621, as discussed above. Video footage 620 may also beassociated with event tag 624, which may identify an intraoperativesurgical event (in this instance an incision) within the surgical phaseoccurring at event location 623. Video footage 620 may further beassociated with event characteristic 625, which may describe one or morecharacteristics of the intraoperative surgical event, such as surgeonskill level, as described in detail above. Each video footage identifiedin the data structure may be associated with more than one footagelocation, phase tag, event location, event tag and/or eventcharacteristic. For example, video footage 610 may be associated withphase tags corresponding to more than one surgical phase (e.g., “Calot'striangle dissection” and “cutting of cystic duct”). Further, eachsurgical phase of a particular video footage may be associated with morethan one event, and accordingly may be associated with more than oneevent location, event tag, and/or event characteristic. It isunderstood, however, that in some embodiments, a particular videofootage may be associated with a single surgical phase and/or event. Itis also understood that in some embodiments, an event may be associatedwith any number of event characteristics, including no eventcharacteristics, a single event characteristic, two eventcharacteristics, more than two event characteristics, and so forth. Somenon-limiting examples of such event characteristics may include skilllevel associated with the event (such as minimal skill level required,skill level demonstrated, skill level of a medical care giver involvedin the event, etc.), time associated with the event (such as start time,end time, etc.), type of the event, information related to medicalinstruments involved in the event, information related to anatomicalstructures involved in the event, information related to medical outcomeassociated with the event, one or more amounts (such as an amount ofleak, amount of medication, amount of fluids, etc.), one or moredimensions (such as dimensions of anatomical structures, dimensions ofincision, etc.), and so forth. Further, it is to be understood that datastructure 600 is provided by way of example and various other datastructures may be used.

Embodiments of the present disclosure may further include enabling auser to access the data structure through selection of a selected phasetag, a selected event tag, and a selected event characteristic of videofootage for display. The user may be any individual or entity that maybe provided access to data stored in the data structure. In someembodiments, the user may be a surgeon or other healthcare professional.For example, a surgeon may access the data structure and/or videofootage associated with the data structure for review or trainingpurposes. In some embodiments, the user may be an administrator, such asa hospital administrator, a manager, a lead surgeon, or other individualthat may require access to video footage. In some embodiments the usermay be a patient, who may be provided access to video footage of his orher surgery. Similarly, the user may be a relative, a guardian, aprimary care physician, an insurance agent, or another representative ofthe patient. The user may include various other entities, which mayinclude, but are not limited to, an insurance company, a regulatoryauthority, a police or investigative authority, a medical association,or any other entity that may be provided access to video footage.Selection by the user may include any means for identifying a particularphase tag, event tag, and/or event characteristic. In some embodiments,selection by the user may occur through a graphical user interface, suchas on a display of a computing device. In another example, the selectionby the user may occur through a touch screen. In an additional example,the selection by the user may occur through voice input, and the voiceinput may be processed using a speech recognition algorithm. In yetanother example, the selection by the user may occur through gestures(such as hand gestures), and the gestures may be analyzed using gesturerecognition algorithms. In some embodiments, the user may not select allthree of the selected phase tag, the selected event tag, or the selectedevent characteristic, but may select a subset of these. For example, theuser may just select an event characteristic and the user may be allowedaccess to information associated with the data structure based on theselected event characteristic.

FIG. 7 is an illustration of exemplary user interface 700 for selectingindexed video footage for display consistent with the disclosedembodiments. User interface 700 may include one or more search boxes710, 720, and 730 for selecting video footage. Search box 710 may allowthe user to select one or more surgical phases to be displayed. In someembodiments, user interface 700 may provide suggested surgical phasesbased on the phase tags include in data structure 600. For example, as auser starts typing in search box 710, user interface 700 may suggestphase tag descriptions to search for based on the characters the userhas entered. In other embodiments, the user may select the phase tagusing radio buttons, checkboxes, a dropdown list, touch interface, orany other suitable user interface feature. Similar to with the phasetags, a user may select video footage based on event tags and eventcharacteristics using search boxes 720 and 730, respectively. Userinterface 700 may also include dropdown buttons 722 and 732 to accessdropdown lists and further filter the results. As shown in FIG. 7 ,selecting dropdown button 732 may allow the user to select an eventcharacteristic based on subcategories of event characteristics. Forexample, a user may select “Surgeon skill level” in the dropdown listassociated with dropdown button 732, which may allow the user to searchbased on a skill level of the surgeon in search box 730. While “Surgeonskill level,” and various other event characteristic subcategories areprovided by way of example, it is understood that a user may select anycharacteristic or property of the surgical procedure. For example, theuser may refine the surgeon skill level based on the surgeon,qualifications, years of experience, and/or any indications of surgicalskill level, as discussed in greater detail below. A user may be enabledto access the data structure by clicking, tapping, or otherwiseselecting search button 740.

Display of video footage may include any process by which one or moreframes of video footage or a portion thereof are presented to the user.In some embodiments, displaying may include electronically transmittingat least a portion of the video footage for viewing by the user. Forexample, displaying the video footage may comprise transmitting at leasta portion of the video footage over a network. In other embodiments,displaying the video footage may include making the video footageavailable to the user by storing the video footage in a locationaccessible to the user or a device being used by the user. In someembodiments, displaying the video footage may comprise causing the videofootage to be played on a visual display device, such as a computer orvideo screen. For example, displaying may include sequentiallypresenting frames associated with the video footage and may furtherinclude presenting audio associated with the video footage.

Some aspects of the present disclosure may include performing a lookupin the data structure of surgical video footage matching the at leastone selected phase tag, selected event tag, and selected eventcharacteristic to identify a matching subset of stored video footage.Performing the lookup may include any process for retrieving data from adata structure. For example, based on the at least one selected phasetag, event tag, and selected event characteristic, a corresponding videofootage or portion of video footage may be identified from the datastructure. A subset of stored video footage may include a singleidentified video footage or multiple identified video footagesassociated with selections of the user. For example, the subset ofstored video footage may include surgical video footage having the atleast one of a phase tag exactly identical to the selected phase tag,event tag exactly identical to the selected event tag, and eventcharacteristic exactly identical to the selected event characteristic.In another example, the subset of stored video footage may includesurgical video footage having the at least one of a phase tag similar(e.g., according to a selected similarity measure) to the selected phasetag, an event tag similar (e.g., according to a selected similaritymeasure) to the selected event tag, and/or an event characteristicsimilar (e.g., according to a selected similarity measure) to theselected event characteristic. In some embodiments, performing thelookup may be triggered by selection of search button 740, as shown inFIG. 7 .

In some exemplary embodiments, identifying a matching subset of storedvideo footage includes using computer analysis to determine a degree ofsimilarity between the matching subset of stored video and the selectedevent characteristic. Accordingly. “matching” may refer to an exactmatch or may refer to an approximate or closest match. In one example,the event characteristic may comprise a numerical value (such as anamount, a dimension, a length, an area, a volume, etc., for example asdescribed above), and the degree of similarity may be based on acomparison of a numerical value included in the selected eventcharacteristic and a corresponding numerical value of a stored video. Inone example, any similarity function (including but not limited toaffinity functions, correlation functions, polynomial similarityfunctions, exponential similarity functions, similarity functions basedon distance, linear functions, non-linear functions, and so forth) maybe used to calculate the degree of similarity. In one example, graphmatching algorithms or hypergraph matching algorithms (such as exactmatching algorithms, inexact matching algorithms) may be used todetermine the degree of similarity. As another illustrative example,video footage associated with a “preparation” phase tag may also beretrieved for phase tags including terms “prep,” “preparing,”“preparatory,” “pre-procedure,” or other similar but not exact matchesthat may refer to a “preparation” phase tag. The degree of similaritymay refer to any measure of how closely the subset of stored videomatches the selected event characteristic. The degree of similarity maybe expressed as a similarity ranking (e.g., on scale of 1-10, 1-100,etc.), as a percentage match, or through any other means of expressinghow closely there is a match. Using computer analysis may include usinga computer algorithm to determine a degree of similarity between theselected event characteristic and the event characteristic of one ormore surgical procedures included in the data structure. In one example,k-Nearest-Neighbors algorithms may be used to identify the most similarentries in the data structure. In one example, the entries of the datastructures, as well as the user inputted event characteristics, may beembedded in a mathematical space (for example, using any dimensionalityreduction or data embedding algorithms), distance between the embeddingof an entry and the user inputted characteristics may be used tocalculate the degree of similarity between the two. Further, in someexamples, the entries nearest to the user inputted characteristics inthe embedded mathematical space may be selected as the most similarentries to the user inputted data in the data structure.

Some aspects of the invention may involve causing the matching subset ofstored video footage to be displayed to the user, to thereby enable theuser to view surgical footage of at least one intraoperative surgicalevent sharing the selected event characteristic, while omitting playbackof video footage lacking the selected event characteristic. Surgicalfootage may refer to any video or video footage, as described in greaterdetail above, capturing a surgical procedure. In some embodiments,causing the matching subset of stored video footage to be displayed maycomprise executing instructions for playing the video. For example, aprocessing device performing the methods described herein may access thematching subset of video footage and may be configured to present thestored video footage to the user on a screen or other display. Forexample, the stored video footage may be displayed in a video playeruser interface, such as in video playback region 410, as discussed infurther detail below with respect to FIG. 4 . In some embodiments,causing the matching subset of stored video footage to be displayed tothe user may include transmitting the stored video footage for display,as described above. For example, the matching subset of video footagemay be transmitted through a network to a computing device associatedwith the user, such as a desktop computer, a laptop computer, a mobilephone, a tablet, smart glasses, heads up display, a training device, orany other device capable of displaying video footage.

Omitting playback may include any process resulting in the video lackingthe selected event characteristic from being presented to the user. Forexample, omitting playback may include designating footage as not to bedisplayed and not displaying that footage. In embodiments where thematching subset of video footage is transmitted, omitting playback mayinclude preventing transmission of video footage lacking the selectedevent characteristic. This may occur by selectively transmitting onlythose portions of footage related to the matching subset; by selectivelytransmitting markers associated with portions of footage related to thematching subset; and/or by skipping over portions of footage unrelatedto the matching subset. In other embodiments, the video footage lackingthe selected event characteristic may be transmitted but may beassociated with one or more instructions not to present the videofootage lacking the selected event characteristic.

According to various exemplary embodiments of the present disclosure,enabling the user to view surgical footage of at least oneintraoperative surgical event that has the selected eventcharacteristic, while omitting playback of portions of selected surgicalevents lacking the selected event characteristic, may includesequentially presenting to the user portions of surgical footage of aplurality of intraoperative surgical events sharing the selected eventcharacteristic, while omitting playback of portions of selected surgicalevents lacking the selected event characteristic. In other words, one ormore portions of video footage may be identified, for example through alookup function in the data structure, as being associated with theselected event characteristic. Enabling the user to view surgicalfootage of the at least one intraoperative surgical event that has theselected event characteristic may include sequentially presenting one ormore of the identified portions to the user. Any portions of videofootage that are not identified may not be presented. In someembodiments, video footage may be selected based on the selected eventtag and the selected phase tag. Accordingly, in embodiments consistentwith the present disclosure, enabling the user to view surgical footageof at least one intraoperative surgical event that has the selectedevent characteristic, while omitting playback of portions of selectedsurgical events lacking the selected event characteristic, may includesequentially presenting to the user portions of surgical footage of aplurality of intraoperative surgical events sharing the selected eventcharacteristic and associated with the selected event tag and theselected phase tag, while omitting playback of portions of selectedsurgical events lacking the selected event characteristic or notassociated the at least one of selected event tag and the selected phasetag.

As mentioned above, the stored event characteristic may include a widevariety of characteristics relating to a surgical procedure. In someexample embodiments, the stored event characteristic may include anadverse outcome of the surgical event. For example, the stored eventcharacteristic may identify whether the event is an adverse event, orwhether it was associated with a complication, including the examplesdescribed in greater detail above. Accordingly, causing the matchingsubset to be displayed may include enabling the user to view surgicalfootage of a selected adverse outcome while omitting playback ofsurgical events lacking the selected adverse outcome. By way of example,in response to a user's desire to see how a surgeon dealt with avascular injury during a laparoscopic procedure, rather than displayingto the user the entire procedure, the user might select the vascularinjury event, after which the system might display only a portion of thevideo footage where the event occurred. The stored event characteristicmay similarly identify outcomes, including desired and/or expectedoutcomes. Examples of such outcomes may include full recovery by thepatient, whether a leak occurred, an amount of leak that occurred,whether the amount of leak was within a selected range, whether thepatient was readmitted after discharge, a length of hospitalizationafter surgery, or any other outcomes that may be associated with thesurgical procedure. In this way, a user may be able to ascertain at thetime of viewing, the long-term impact of a particular technique.Accordingly, in some embodiments, the stored event characteristic mayinclude these or other outcomes, and causing the matching subset to bedisplayed may include enabling the user to view surgical footage of theselected outcome while omitting playback of surgical events lacking theselected outcome.

In some embodiments, the stored event characteristic may include asurgical technique. Accordingly, the stored event characteristic mayidentify whether a particular technique is performed. For example, theremay be multiple techniques that may be applied at a particular stage ofsurgery and the event characteristic may identify which technique isbeing applied. In this way, a user interested in learning a particulartechnique might be able to filter video results so that only proceduresusing the specified technique are displayed. Causing the matching subsetto be displayed may include enabling the user to view surgical footageof a selected surgical technique while omitting playback of surgicalfootage not associated with the selected surgical technique. Forexample, the user may be enabled to view in sequence, non-sequentialportions of video captured from either the same surgery or fromdifferent surgeries. In some embodiments, the stored eventcharacteristic may include an identity of a specific surgeon. Forexample, the event characteristic may include an identity of aparticular surgeon performing the surgical procedure. The surgeon may beidentified based on his or her name, an identification number (e.g.,employee number, medical registration number, etc.) or any other form ofidentity. In some embodiments, the surgeon may be identified based onrecognizing representations of the surgeon in the captured video. Forexample, various facial and/or voice recognition techniques may be used,as discussed above. In this way, if a user wishes to study a techniqueof a particular surgeon, the user may be enabled to do so. For example,causing the matching subset to be displayed may include enabling theuser to view footage exhibiting an activity by a selected surgeon whileomitting playback of footage lacking activity by the selected surgeon.Thus for example, if multiple surgeons participate in the same surgicalprocedure, a user may choose to view only the activities of a subset ofthe team.

In some embodiments, the event characteristic may also be associatedwith other healthcare providers or healthcare professionals who may beinvolved in the surgery. In some examples, a characteristic associatedwith a healthcare provider may include any characteristic of ahealthcare provider involved in the surgical procedure. Somenon-limiting examples of such healthcare providers may include the titleof any member of the surgical team, such as surgeons, anesthesiologists,nurses, Certified Registered Nurse Anesthetist (CRNA), surgical tech,residents, medical students, physician assistants, and so forth.Additional non-limiting examples of such characteristics may includecertification, level of experience (such as years of experience, pastexperience in similar surgical procedures, past success rate in similarsurgical procedures, etc.), demographic characteristics (such as age),and so forth.

In other embodiments, the stored event characteristic may include a timeassociated with the particular surgical procedure, surgical phase, orportion thereof. For example, the stored event characteristic mayinclude a duration of the event. Causing the matching subset to bedisplayed may include enabling the user to view footage exhibitingevents of selected durations while omitting playback of footage ofevents of different durations. In this way, for example, a user whomight wish to view a particular procedure completed more quickly thanthe norm, might set a time threshold to view specified procedurescompleted within that threshold. In another example, a user who mightwish to view more complex events may set a time threshold to viewprocedures including events lasting longer than a selected threshold, orthe procedures including events that lasted the longest of a selectedgroup of events. In other embodiments, the stored event characteristicmay include a starting time of the event, an ending time of the event,or any other time indicators. Causing the matching subset to bedisplayed may include enabling the user to view footage exhibitingevents from selected times within the particular surgical procedure,within the phase associated with the event, or within the selectedportion of the particular surgical procedure, while omitting playback offootage of events associated with different times.

In another example, the stored event characteristic may include apatient characteristic. The term “patient characteristic” refers to anyphysical, sociological, economical, demographical or behavioralcharacteristics of the patient, and to characteristics of the medicalhistory of the patient. Some non-limiting examples of such patientcharacteristics may include age, gender, weight, height, Body Mass Index(BMI), menopausal status, typical blood pressure, characteristics of thepatient genome, educational status, level of education, socio-economicstatus, level of income, occupation, type of insurance, health status,self-rated health, functional status, functional impairment, duration ofdisease, severity of disease, number of illnesses, illnesscharacteristics (such as type of illness, size of tumor, histologygrade, number of infiltrated lymph nodes, etc.), utilization of healthcare, number of medical care visits, medical care visit intervals,regular source of medical care, family situation, marital status, numberof children, family support, ethnicity, race, acculturation, religious,type of religion, native language, characteristics of past medical testperformed on the patient (such as type of test, time of test, results oftest, etc.), characteristics of past medical treatments performed on thepatient (such as type of treatment, time of treatment, results oftreatment, etc.), and so forth. Some non-limiting examples of suchmedical tests may include blood tests, urine tests, stool tests, medicalimaging (such as ultrasonography, angiography, Magnetic ResonanceImaging (MRI), Computed Tomography (CT), X-ray, electromyography,Positron Emission Tomography (PET), etc.), physical examination,electrocardiography, amniocentesis, pap test, skin allergy tests,endoscopy, biopsy, pathology, blood pressure measurements, oxygensaturation test, pulmonary function test, and so forth. Somenon-limiting examples of such medical treatments may include medication,dietary treatment, surgery, radiotherapy, chemotherapy, physicaltherapy, psychological therapy, blood transfusion, infusion, and soforth. Accordingly, causing the matching subset to be displayed mayinclude enabling the user to view footage of patients exhibiting aselected patient characteristic while omitting playback of footage ofpatients lacking the selected patient characteristic.

In some embodiments, the selected physical patient characteristic mayinclude a type of anatomical structure. As used herein, an anatomicalstructure may be any particular part of a living organism. For example,an anatomical structure may include any particular organ, tissue, cell,or other structures of the patient. In this way, if for example, a userwishes to observe video relating to surgery on a pleura sack in a lung,that portion of footage may be presented while other non-relatedportions may be omitted. The stored event characteristic may includevarious other patient characteristics, such as the patient'sdemographics, medical condition, medical history, previous treatments,or any other relevant patient descriptor. This can enable a viewer toview surgical procedures on patients matching very particularcharacteristics (e.g., 70-75 year old Caucasian, with coronary heartdisease who previously had bypass surgery. In this way, video of one ormore patients matching those specific criteria might be selectivelypresented to the user.

In yet another example, the stored event characteristic may include aphysiological response. As used herein, the term “physiologicalresponse” refers to any physiological change that may have occurred inreaction to an event within a surgical procedure. Some non-limitingexamples of such physiological changes may include change in bloodpressure, change in oxygen saturation, change in pulmonary functions,change in respiration rate, change in blood composition (countchemistry, etc.), bleeding, leakage, change in blood flow to a tissue,changing in a condition of a tissue (such as change in color, shape,structural condition, functional condition, etc.), change in bodytemperature, a change in brain activity, a change in perspiration, orany other physical change in response to the surgical procedure. In thisway, a user might be able to prepare for eventualities that might occurduring a surgical procedure by selectively viewing those eventualities(and omitting playback of non-matching eventualities.).

In some examples, the event characteristic may include a surgeon skilllevel. The skill level may include any indication of the surgeon'srelative abilities. In some embodiments, the skill level may include ascore reflecting the surgeon's experience or proficiency in performingthe surgical procedure or specific techniques within the surgicalprocedure. In this way, a user can compare, by selecting different skilllevels how surgeons of varying experience handle the same procedure. Insome embodiments the skill level may be determined based on the identityof a surgeon, either determined via data entry (manually inputting thesurgeon's ID) or by machine vision. For example, the disclosed methodsmay include analysis of the video footage to determine an identity ofthe surgeon through biometric analysis (e.g., face, voice, etc.) andidentify a predetermined skill level associated with that surgeon. Thepredetermined skill level may be obtained by accessing a databasestoring skill levels associated with particular surgeons. The skilllevel may be based on past performances of the surgeon, a type and/orlevel of training or education of the surgeon, a number of surgeries thesurgeon has performed, types of surgeries surgeon has performed,qualifications of the surgeon, a level of experience of the surgeon,ratings of the surgeon from patients or other healthcare professionals,past surgical outcomes, past surgical outcomes and complications, or anyother information relevant to assessing the skill level of a healthcareprofessional. In some embodiments, the skill level may be determinedautomatically based on computer analysis of the video footage. Forexample, the disclosed embodiments, may include analyzing video footagecapturing performance of a procedure, performance of a particulartechnique, a decision made by the surgeon, or similar events. The skilllevel of the surgeon may then be determined based on how well thesurgeon performs during the event, which may be based on timeliness,effectiveness, adherence to a preferred technique, the lack of injury oradverse effects, or any other indicator of skill that may be gleanedfrom analyzing the footage.

In some embodiments, the skill level may be a global skill levelassigned to each surgeon or may be in reference to specific events. Forexample, a surgeon may have a first skill level with regard to a firsttechnique or procedure and may have a second skill level with regard toa different technique or procedure. The skill level of the surgeon mayalso vary throughout an event, technique and/or procedure. For example,a surgeon may act at a first skill level within a first portion of thefootage but may act at a second skill level at a second portion of thefootage. Accordingly, the skill level may be a skill level associatedwith a particular location of the footage. The skill level also may be aplurality of skill levels during an event or may be an aggregation ofthe plurality of skill levels during the event, such as an averagevalue, a rolling average, or other forms of aggregation. In someembodiments, the skill level may be a general required skill level forperforming the surgical procedure, the surgical phase, and/or theintraoperative surgical event and may not be tied to a particularsurgeon or other healthcare professional. The skill level may beexpressed in various ways, including as a numerical scale (e.g., 1-10,1-100, etc.), as a percentage, as a scale of text-based indicators(e.g., “highly skilled,” “moderately skilled,” “unskilled,” etc.) or anyother suitable format for expressing the skill of a surgeon. While theskill level is described herein as the skill level of a surgeon, in someembodiments the skill level may be associated with another healthcareprofessional, such as a surgical technician, a nurse, a physician'sassistant, an anesthesiologist, a doctor, or any other healthcareprofessional.

Embodiments of the present disclosure may further include accessingaggregate data related to a plurality of surgical procedures similar tothe particular surgical procedure. Aggregate data may refer to datacollected and/or combined from multiple sources. The aggregate data maybe compiled from multiple surgical procedures having some relation tothe particular surgical procedure. For example, a surgical procedure maybe considered similar to the particular surgical procedure if itincludes the same or similar surgical phases, includes the same orsimilar intraoperative events, or is associated with the same or similartags or properties (e.g., event tags, phase tags, event characteristics,or other tags.).

The present disclosure may further include presenting to the userstatistical information associated with the selected eventcharacteristic. Statistical information may refer to any informationthat may be useful to analyze multiple surgical procedures together.Statistical information may include, but is not limited to, averagevalues, data trends, standard deviations, variances, correlations,causal relations, test statistics (including t statistics, chi-squaredstatistics, f statistics, or other forms of test statistics), orderstatistics (including sample maximum and minimum), graphicalrepresentations (e.g., charts, graphs, plots, or other visual orgraphical representations), or similar data. As an illustrative example,in embodiments where the user selects an event characteristic includingthe identity of a particular surgeon, the statistical information mayinclude the average duration in which the surgeon performs the surgicaloperation (or phase or event of the surgical operation), the rate ofadverse or other outcomes the surgeon, the average skill level at whichthe surgeon performs an intraoperative event, or similar statisticalinformation. A person of ordinary skill in the art would appreciateother forms of statistical information that may be presented accordingto the disclosed embodiments.

FIGS. 8A and 8B are flowcharts illustrating an example process 800 forvideo indexing consistent with the disclosed embodiments. Process 800may be performed by a processing device, such as at least one processor.For example, the at least one processor may include one or moreintegrated circuits (IC), including application-specific integratedcircuit (ASIC), microchips, microcontrollers, microprocessors, all orpart of a central processing unit (CPU), graphics processing unit (GPU),digital signal processor (DSP), field-programmable gate array (FPGA),server, virtual server, or other circuits suitable for executinginstructions or performing logic operations. The instructions executedby at least one processor may, for example, be pre-loaded into a memoryintegrated with or embedded into the controller or may be stored in aseparate memory. The memory may include a Random Access Memory (RAM), aRead-Only Memory (ROM), a hard disk, an optical disk, a magnetic medium,a flash memory, other permanent, fixed, or volatile memory, or any othermechanism capable of storing instructions. In some embodiments, the atleast one processor may include more than one processor. Each processormay have a similar construction or the processors may be of differingconstructions that are electrically connected or disconnected from eachother. For example, the processors may be separate circuits orintegrated in a single circuit. When more than one processor is used,the processors may be configured to operate independently orcollaboratively. The processors may be coupled electrically,magnetically, optically, acoustically, mechanically or by other meansthat permit them to interact.

In some embodiments, a non-transitory computer readable medium maycontain instructions that when executed by a processor cause theprocessor to perform process 800. At step 802, process 800 may includeaccessing video footage to be indexed, the video footage to be indexedincluding footage of a particular surgical procedure. The video footagemay be accessed from a local memory, such as a local hard drive, or maybe accessed from a remote source, for example, through a networkconnection. In another example, the video footage may be captured usingone or more image sensors, or generated by another process. At step 804,process 800 may include analyzing the video footage to identify a videofootage location associated with a surgical phase of the particularsurgical procedure. As discussed above, the location may be associatedwith a particular frame, a range of frames, a time index, a time range,or any other location identifier.

Process 800 may include generating a phase tag associated with thesurgical phase, as shown in step 806. This may occur, for example,through video content analysis (VCA), using techniques such as one ormore of video motion detection, video tracking, shape recognition,object detection, fluid flow detection, equipment identification,behavior analysis, or other forms of computer aided situationalawareness. When learned characteristics associated with a phase areidentified in the video, a tag may be generated demarcating that phase.The tag may include, for example, a predefined name for the phase. Atstep 808, process 800 may include associating the phase tag with thevideo footage location. The phase tag may indicate, for example, thatthe identified video footage location is associated with the surgicalphase of the particular surgical procedure. At step 810, process 800 mayinclude analyzing the video footage using one or more of the VCAtechniques described above, to identify an event location of aparticular intraoperative surgical event within the surgical phase.Process 800 may include associating an event tag with the event locationof the particular intraoperative surgical event, as shown at step 812.The event tag may indicate, for example, that the video footage isassociated with the surgical event at the event location. As with thephase tag, the event tag may include a predefined name for the event. Atstep 814, in FIG. 8B, process 800 may include storing an eventcharacteristic associated with the particular intraoperative surgicalevent. As discussed in greater detail above, the event characteristicmay include an adverse outcome of the surgical event, a surgicaltechnique, a surgeon skill level, a patient characteristic, an identityof a specific surgeon, a physiological response, a duration of theevent, or any other characteristic or property associated with theevent. The event characteristic may be manually determined (for example,inputted by a viewer), or may be determined automatically throughartificial intelligence applied to machine vision, for example asdescribed above. In one example, the event characteristic may includeskill level (such as minimal skill level required, skill leveldemonstrated during the event, etc.), a machine learning model may betrained using training example to determine such skill levels fromvideos, and the trained machine learning model may be used to analyzethe video footage to determine the skill level. An example of suchtraining example may include a video clip depicting an event togetherwith a label indicating the corresponding skill level. In anotherexample, the event characteristic may include time relatedcharacteristics of the event (such as start time, end time, duration,etc.), and such time related characteristics may be calculated byanalyzing the interval in the video footage corresponding to the event.In yet another example, the event characteristic may include an eventtype, a machine learning model may be trained using training examples todetermine event types from videos, and the trained machine learningmodel may be used to analyze the video footage and determine the eventtype. An example of such training example may include a video clipdepicting an event together with a label indicating the event type. Inan additional example, the event characteristic may include informationrelated to a medical instrument involved in the event (such as type ofmedical instrument, usage of the medical instrument, etc.), a machinelearning model may be trained using training examples to identify suchinformation related to medical instruments from videos, and the trainedmachine learning model may be used to analyze the video footage anddetermine the information related to a medical instrument involved inthe event. An example of such training example may include video clipdepicting an event including a usage of a medical instrument, togetherwith a label indicative of the information related to the medicalinstrument. In yet another example, the event characteristic may includeinformation related to an anatomical structure involved in the event(such as type of the anatomical structure, condition of the anatomicalstructure, change occurred to the anatomical structure in relation tothe event, etc.), a machine learning model may be trained using trainingexample to identify such information related to anatomical structuresfrom videos, and the trained machine learning model may be used toanalyze the video footage and determine the information related to theanatomical structure involved in the event. An example of such trainingexample may include a video clip depicting an event involving ananatomical structure, together with a label indicative of informationrelated to the anatomical structure. In an additional example, the eventcharacteristic may include information related to a medical outcomeassociated with the event, a machine learning model may be trained usingtraining example to identify such information related to medicaloutcomes from videos, and the trained machine learning model may be usedto analyze the video footage and determine the information related tothe medical outcome associated with the event. An example of suchtraining example may include a video clip depicting a medical outcome,together with a label indicative of the medical outcome.

At step 816, process 800 may include associating at least a portion ofthe video footage of the particular surgical procedure with at least oneof the phase tag, the event tag, and the event characteristic in a datastructure. In this step, the various tags are associated with the videofootage to permit the tags to be used to access the footage. Aspreviously described, various data structures may be used to storerelated data in an associated manner.

At step 818, process 800 may include enabling a user to access the datastructure through selection of at least one of a selected phase tag, aselected event tag, and a selected event characteristic of video footagefor display. In some embodiments, the user may select the selected phasetag, selected event tag, and selected event characteristic through auser interface of a computing device, such as user interface 700 shownin FIG. 7 . For example, data entry fields, drop down menus, icons, orother selectable items may be provided to enable a user to select asurgical procedure, the phase of the procedure, an event within aprocedure and a characteristic of the procedure and patient. At step820, process 800 may include performing a lookup in the data structureof surgical video footage matching the at least one selected phase tag,selected event tag, and selected event characteristic to identify amatching subset of stored video footage. At step 822, process 800 mayinclude causing the matching subset of stored video footage to bedisplayed to the user, to thereby enable the user to view surgicalfootage of at least one intraoperative surgical event sharing theselected event characteristic, while omitting playback of video footagelacking the selected event characteristic. Through this filtering, theuser may be able to quickly view only those video segments correspondingto the user's interest, while omitting playback of large volumes ofvideo data unrelated to the user's interest.

When preparing for a surgical procedure, it may be beneficial for asurgeon to review video footage of surgical procedures having similarsurgical events. It may be too time consuming, however, for a surgeon toview the entire video or to skip around to find relevant portions of thesurgical footage. Therefore, there is a need for unconventionalapproaches that efficiently and effectively enable a surgeon to view asurgical video summary that aggregates footage of relevant surgicalevents while omitting other irrelevant footage.

Aspects of this disclosure may relate to generating surgical summaryfootage, including methods, systems, devices, and computer readablemedia. For example, footage of one surgical procedure may be comparedwith that of previously analyzed procedures to identify and tag relevantintraoperative surgical events. A surgeon may be enabled to watch asummary of a surgery that aggregates the intraoperative surgical events,while omitting much of the other irrelevant footage. For ease ofdiscussion, a method is described below, with the understanding thataspects of the method apply equally to systems, devices, and computerreadable media. For example, some aspects of such a method may occurelectronically over a network that is either wired, wireless, or both.Other aspects of such a method may occur using non-electronic means. Ina broadest sense, the method is not limited to particular physicaland/or electronic instrumentalities, but rather may be accomplishedusing many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve accessingparticular surgical footage containing a first group of framesassociated with at least one intraoperative surgical event. Surgicalfootage may refer to any video, group of video frames, or video footageincluding representations of a surgical procedure. For example, thesurgical footage may include one or more video frames captured during asurgical operation. Accessing the surgical footage may includeretrieving video from a storage location, such as a memory device. Thesurgical footage may be accessed from a local memory, such as a localhard drive, or may be accessed from a remote source, for example,through a network connection. As described in greater detail above,video may include any form of recorded visual media including recordedimages and/or sound. The video may be stored as a video file such as anAudio Video Interleave (AVI) file, a Flash Video Format (FLV) file,QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), a Windows MediaVideo (WMV) file, a Material Exchange Format (MXF) file, or any othersuitable video file formats. Additionally or alternatively, in someexamples accessing particular surgical footage may include capturing theparticular surgical footage using one or more image sensors.

As described above, the intraoperative surgical event may be any eventor action that is associated with a surgical procedure or phase. A framemay refer to one of a plurality of still images which compose a video.The first group of frames may include frames that were captured duringthe interoperative surgical event. For example, the particular surgicalfootage may depict a surgical procedure performed on a patient andcaptured by at least one image sensor in an operating room. The imagesensors may include, for example, cameras 115, 121, and 123, and/or 125located in operating room 101. In some embodiments, the at least oneimage sensor may be at least one of above an operating table in theoperating room or within the patient. For example, the image sensor maybe located above the patient, or may be located within a surgicalcavity, organ, or vasculature of the patient, as described above. Thefirst group of frames may include representations of the intraoperativesurgical event, including anatomical structures, surgical tools,healthcare professionals performing the intraoperative surgical event,or other visual representations of the intraoperative surgical event. Insome embodiments, however, some or all of the frames may not containrepresentations of the intraoperative surgical event, but may beotherwise associated with the event (e.g., captured while the event wasbeing performed, etc.).

Consistent with the present disclosure, the particular surgical footagemay contain a second group of frames not associated with surgicalactivity. For example, surgical procedures may involve extensive periodsof downtime, where significant surgical activity is not taking place andwhere there would be no material reason for review of the footage.Surgical activity may refer to any activities that are performed inrelation to a surgical procedure. In some embodiments, surgical activitymay broadly refer to any activities associated with the surgicalprocedure, including preoperative activity, perioperative activity,intraoperative activity, and/or postoperative activity. Accordingly, thesecond group of frames may include frames not associated with any suchactivities. In other embodiments, surgical activity may refer to anarrower set of activity, such as physical manipulation of organs ortissues of the patient being performed by the surgeon. Accordingly, thesecond group of frames may include various activities associated withpreparation, providing anesthesia, monitoring vital signs, gathering orpreparing surgical tools, discussion between healthcare professionals,or other activities that may not be considered surgical activity.

In accordance with the present disclosure, the methods may includeaccessing historical data based on historical surgical footage of priorsurgical procedures. Historical data may refer to data of any formatthat was recorded and/or stored previously. In some embodiments, thehistorical data may be one or more video files including the historicalsurgical footage. For example, the historical data may include a seriesof frames captured during the prior surgical procedures. This historicaldata is not limited to video files, however. For example, the historicaldata may include information stored as text representing at least oneaspect of the historical surgical footage. For example, the historicaldata may include a database of information summarizing or otherwisereferring to historical surgical footage. In another example, thehistorical data may include information stored as numerical valuesrepresenting at least one aspect of the historical surgical footage. Inan additional example, the historical data may include statisticalinformation and/or statistical model based on an analysis of thehistorical surgical footage. In yet another example, the historical datamay include a machine learning model trained using training examples,and the training examples may be based on the historical surgicalfootage. Accessing the historical data may include receiving thehistorical data through an electronic transmission, retrieving thehistorical data from storage (e.g., a memory device), or any otherprocess for accessing data. In some embodiments, the historical data maybe accessed from the same resource as the particular surgical footagediscussed above. In other embodiments, the historical data may beaccessed from a separate resource. Additionally or alternatively,accessing the historical data may include generating the historicaldata, for example by analyzing the historical surgical footage of priorsurgical procedures or by analyzing data based on the historicalsurgical footage of prior surgical procedures.

In accordance with embodiments of the present disclosure, the historicaldata may include information that distinguishes portions of surgicalfootage into frames associated with intraoperative surgical events andframes not associated with surgical activity. The information maydistinguish the portions of surgical footage in various ways. Forexample, in connection with historical surgical footage, framesassociated with surgical and non-surgical activity may already have beendistinguished. This may have previously occurred, for example, throughmanual flagging of surgical activity or through training of anartificial intelligence engine to distinguish between surgical andnon-surgical activity. The historical information may identify, forexample, a set of frames (e.g., using a starting frame number, a numberof frames, an end frame number, etc.) of the surgical footage. Theinformation may also include time information, such as a begintimestamp, an end timestamp, a duration, a timestamp range, or otherinformation related to timing of the surgical footage. In one example,the historical data may include various indicators and/or rules thatdistinguish the surgical activity from non-surgical activity. Somenon-limiting examples of such indicators and/or rules are discussedbelow. In another example, the historical data may include a machinelearning model trained to identify portions of videos corresponding tosurgical activity and/or portions of videos corresponding tonon-surgical activity, for example based on the historical surgicalfootage.

Various indicators may be used to distinguish the surgical activity fromnon-surgical activity—either manually, semi-manually, of automatically(for example, via machine learning). For example, in some embodiments,the information that distinguishes portions of the historical surgicalfootage into frames associated with an intraoperative surgical event mayinclude an indicator of at least one of a presence or a movement of asurgical tool. A surgical tool may be any instrument or device that maybe used during a surgical procedure, which may include, but is notlimited to, cutting instruments (such as scalpels, scissors, saws,etc.), grasping and/or holding instruments (such as Billroth's clamps,hemostatic “mosquito” forceps, atraumatic hemostatic forceps, Deschamp'sneedle, Hopfner's hemostatic forceps, etc.), retractors (such asFarabefs Cshaped laminar hook, blunt-toothed hook, sharp-toothed hook,grooved probe, tamp forceps, etc.), tissue unifying instruments and/ormaterials (such as needle holders, surgical needles, staplers, clips,adhesive tapes, mesh, etc.), protective equipment (such as facial and/orrespiratory protective equipment, headwear, footwear, gloves, etc.),laparoscopes, endoscopes, patient monitoring devices, and so forth. Avideo or image analysis algorithm, such as those described above withrespect to video indexing, may be used to detect the presence and/ormotion of the surgical tool within the footage. In some examples, ameasure of motion of the surgical tool may be calculated, and thecalculated measure of motion may be compared with a selected thresholdto distinguish the surgical activity from non-surgical activity. Forexample, the threshold may be selected based on a type of surgicalprocedure, based on time of or within the surgical procedure, based on aphase of the surgical procedure, based on parameters determined byanalyzing video footage of the surgical procedure, based on parametersdetermined by analyzing the historical data, and so forth. In someexamples, signal processing algorithms may be used to analyze calculatedmeasures of motion for various times within the video footage of thesurgical procedure to distinguish the surgical activity fromnon-surgical activity. Some non-limiting examples of such signalprocessing algorithms may include machine learning based signalprocessing algorithms trained using training examples to distinguish thesurgical activity from non-surgical activity, artificial neural networks(such as recursive neural networks, long short-term memory neuralnetworks, deep neural networks, etc.) configured to distinguish thesurgical activity from non-surgical activity, Markov models, Viterbimodels, and so forth.

In some exemplary embodiments, the information that distinguishesportions of the historical surgical footage into frames associated withan intraoperative surgical event may include detected tools andanatomical features in associated frames. For example, the disclosedmethods may include using an image and/or video analysis algorithm todetect tools and anatomical features. The tools may include surgicaltools, as described above, or other nonsurgical tools. The anatomicalfeatures may include anatomical structures (as defined in greater detailabove) or other parts of a living organism. The presence of both asurgical tool and an anatomical structure detected in one or moreassociated frames, may serve as an indicator of surgical activity, sincesurgical activity typically involves surgical tools interacting withanatomical structures. For example, in response to a detection of afirst tool in a group of frames, the group of frames may be determinedto be associated with an intraoperative surgical event, while inresponse to no detection of the first tool in the group of frames, thegroup of frames may be Identified as not associated with theintraoperative surgical event. In another example, in response to adetection of a first anatomical feature in a group of frames, the groupof frames may be determined to be associated with an intraoperativesurgical event, while in response to no detection of the firstanatomical feature in the group of frames, the group of frames may beidentified as not associated with the intraoperative surgical event. Insome examples, video footage may be further analyzed to detectinteraction between the detected tools and anatomical features, anddistinguishing the surgical activity from non-surgical activity may bebased on the detected interaction. For example, in response to adetection of a first interaction in a group of frames, the group offrames may be determined to be associated with an intraoperativesurgical event, while in response to no detection of the firstinteraction in the group of frames, the group of frames may beidentified as not associated with the intraoperative surgical event. Insome examples, video footage may be further analyzed to detect actionsperformed by the detected tools, and distinguishing the surgicalactivity from non-surgical activity may be based on the detectedactions. For example, in response to a detection of a first action in agroup of frames, the group of frames may be determined to be associatedwith an intraoperative surgical event, while in response to no detectionof the first action in the group of frames, the group of frames may beidentified as not associated with the intraoperative surgical event. Insome examples, video footage may be further analyzed to detect changesin the condition of anatomical features, and distinguishing the surgicalactivity from non-surgical activity may be based on the detectedchanges. For example, in response to a detection of a first change in agroup of frames, the group of frames may be determined to be associatedwith an intraoperative surgical event, while in response to no detectionof the first change in the group of frames, the group of frames may beidentified as not associated with the intraoperative surgical event.

Some aspects of the invention may involve distinguishing in theparticular surgical footage the first group of frames from the secondgroup of frames based on the information of the historical data. Forexample, the information may provide context that is useful indetermining which frames of the particular surgical footage areassociated with intraoperative events and/or surgical activity. In someembodiments, distinguishing in the particular surgical footage the firstgroup of frames from the second group of frames may involve the use of amachine learning algorithm. For example, a machine learning model may betrained to identify intraoperative events and/or surgical activity usingtraining examples based on the information of the historical data.

In accordance with the present disclosure, the first and second group offrames may be distinguished by analyzing the surgical footage toidentify information similar to the information of the historical data.FIG. 9 is a flowchart illustrating an example process 900 fordistinguishing the first group of frames from the second group offrames. It is to be understood that process 900 is provided by way ofexample. A person of ordinary skill would appreciate various otherprocesses for distinguishing the first group of frames from the secondgroup, consistent with this disclosure. At step 910, process 900 mayinclude analyzing the particular surgical footage to detect a medicalinstrument. A medical instrument may refer to any tool or device usedfor treatment of a patient, including surgical tools, as describedabove. In addition to the surgical tools listed above, medicalinstruments may include, but are not limited to stethoscopes, gauzesponges, catheters, cannulas, defibrillators, needles, trays, lights,thermometers, pipettes or droppers, oxygen masks and tubes, or any othermedical utensils. For example, a machine learning model may be trainedusing training examples to detect medical instruments in images and/orvideos, and the trained machine learning model may be used to analyzethe particular surgical footage and detect the medical instrument. Anexample of such training example may include a video and/or an image ofa surgical procedure, together with a label indicating the presence ofone or more particular medical instruments in the video and/or in theimage, or together with a label indicating an absence of particularmedical instruments in the video and/or in the image.

At step 920, process 900 may include analyzing the particular surgicalfootage to detect an anatomical structure. The anatomical structure maybe any organ, part of an organ, or other part of a living organism, asdiscussed above. One or more video and/or image recognition algorithms,as described above, may be used to detect the medical instrument and/oranatomical structure. For example, a machine learning model may betrained using training examples to detect anatomical structures inimages and/or videos, and the trained machine learning model may be usedto analyze the particular surgical footage and detect the anatomicalstructure. An example of such training example may include a videoand/or an image of a surgical procedure, together with a labelindicating the presence of one or more particular anatomical structuresin the video and/or in the image, or together with a label indicating anabsence of particular anatomical structures in the video and/or in theimage.

At step 930, process 900 may include analyzing the video to detect arelative movement between the detected medical instrument and thedetected anatomical structure. Relative movement may be detected using amotion detection algorithm, for example, based on changes in pixelsbetween frames, optical flow, or other forms of motion detectionalgorithms. For example, motion detection algorithms may be used toestimate the motion of the medical instrument in the video and toestimate the motion of the anatomical structure in the video, and theestimated motion of the medical instrument may be compared with theestimated motion of the anatomical structure to determine the relativemovement. At step 940, process 900 may include distinguishing the firstgroup of frames from the second group of frames based on the relativemovement, wherein the first group of frames includes surgical activityframes and the second group of frames includes non surgical activityframes. For example, in response to a first relative movement pattern ina group of frames, it may be determined that the group of framesincludes surgical activity, while in response to a detection of a secondrelative movement pattern in the group of frames, the group of framesmay be identified as not including non surgical activity frames.Accordingly, presenting an aggregate of the first group of frames maythereby enable a surgeon preparing for surgery to omit the non-surgicalactivity frames during a video review of the abridged presentation. Insome embodiments, omitting the non-surgical activity frames may includeomitting a majority of frames that capture non-surgical activity. Forexample, not all frames that capture non-surgical activity may beomitted, such as frames that immediately precede or followintraoperative surgical events, frames capturing non-surgical activitythat provides context to intraoperative surgical events, or any otherframes that may be relevant to a user.

In some exemplary embodiments of the present disclosure, distinguishingthe first group of frames from the second group of frames may further bebased on a detected relative position between the medical instrument andthe anatomical structure. The relative position may refer to a distancebetween the medical instrument and the anatomical structure, anorientation of the medical instrument relative to the anatomicalstructure, or the location of the medical instrument relative to theanatomical structure. For example, the relative position may beestimated based on a relative position of the detected medicalinstrument and anatomical structure within one or more frames of thesurgical footage. For example, the relative position may include adistance (for example, in pixels, in real world measurements, etc.), adirection, a vector, and so forth. In one example, object detectionalgorithms may be used to determine a position of the medicalinstrument, and to determine a position of the anatomical structure, andthe two determined positions may be compared to determine the relativeposition. In one example, in response to a first relative position in agroup of frames, it may be determined that the group of frames includessurgical activity, while in response to a detection of a second relativeposition in the group of frames, the group of frames may be identifiedas non surgical activity frames. In another example, the distancebetween the medical instrument and the anatomical structure may becompared with a selected threshold, and distinguishing the first groupof frames from the second group of frames may further be based on aresult of the comparison. For example, the threshold may be selectedbased on the type of the medical instrument, the type of the anatomicalstructure, the type of the surgical procedure, and so forth. In otherembodiments, distinguishing the first group of frames from the secondgroup of frames may further be based on a detected interaction betweenthe medical instrument and the anatomical structure. An interaction mayinclude any action by the medical instrument that may influence theanatomical structure, or vice versa. For example, the interaction mayinclude a contact between the medical instrument and the anatomicalstructure, an action by the medical instrument on the anatomicalstructure (such as cutting, clamping, applying pressure, scraping,etc.), a reaction by the anatomical structure (such as a reflex action),or any other form of interaction. For example, a machine learning modelmay be trained using training examples to detect interactions betweenmedical instruments and anatomical structures from videos, and thetrained machine learning model may be used to analyze the video footageand detect the interaction between the medical instrument and theanatomical structure. An example of such training example may include avideo clip of a surgical procedure, together with a label indicating thepresence of particular interactions between medical instruments andanatomical structures in the video clip, or together with a labelindicating the absence of particular interactions between medicalinstruments and anatomical structures in the video clip.

Some aspects of the present disclosure may involve, upon request of auser, presenting to the user an aggregate of the first group of framesof the particular surgical footage, while omitting presentation to theuser of the second group of frames. The aggregate of the first group offrames may be presented in various forms. In some embodiments, theaggregate of the first group of frames may include a video file. Thevideo file may be a compilation of video clips including the first groupof frames. In some embodiments the user may be presented each of thevideo clips separately, or may be presented a single compiled video. Insome embodiments a separate video file may be generated for theaggregate of the first group of frames. In other embodiments, theaggregate of the first group of frames my include instructions foridentifying frames to be included for presentation, and frames to beomitted. Execution of the instructions may appear to the user as if acontinuous video has been generated. Various other formats may also beused, including presenting the first group of frames as still images.

Presenting may include any process for delivering the aggregate to theuser. In some embodiments, this may include causing the aggregate to beplayed on a display, such as a computer screen or monitor, a projector,a mobile phone display, a tablet, a smart device, or any device capableof displaying images and/or audio. Presenting may also includetransmitting the aggregate of the first group of frames to the user orotherwise making it accessible to the user. For example, the aggregateof the first group of frames may be transmitted through a network to acomputing device of the user. As another example, the location of theaggregate of the first group of frames may be shared with the user. Thesecond group of frames may be omitted by not including the second groupof frames in the aggregate. For example, if the aggregate is presentedas a video, video clips comprising the second group of frames may not beincluded in the video file. The first group of frames may be presentedin any order, including chronological order. In some instances, it maybe logical to present at least some of the first group of frames innon-chronological order. In some embodiments, the aggregate of the firstgroup of frames may be associated with more than one intraoperativesurgical event. For example, a user may request to view a plurality ofintraoperative surgical events in the particular surgical footage.Presenting to the user an aggregate of the first group of frames mayinclude displaying the first group frames in chronological order withchronological frames of the second group omitted.

The user may be any individual or entity that may require access tosurgical summary footage. In some embodiments, the user may be a surgeonor other healthcare professional. For example, a surgeon may requestsurgical summary footage for review or training purposes. In someembodiments the user may be an administrator, a manager, a lead surgeon,insurance company personnel, a regulatory authority, a police orinvestigative authority, or any other entity that may require access tosurgical footage. Various other examples of users are provided above inreference to video indexing techniques. The user may submit the requestthrough a computer device, such as a laptop, a desktop computer, amobile phone, a tablet, smart glasses or any other form of computingdevice capable of submitting requests. In some embodiments, the requestmay be received electronically through a network and the aggregate maybe presented based on receipt of the request.

In some exemplary embodiments, the request of the user may include anindication of at least one type of intraoperative surgical event ofinterest and the first group of frames may depict at least oneintraoperative surgical event of the at least one type of intraoperativesurgical event of interest. The type of the intraoperative surgicalevent may be any category in which the intraoperative surgical event maybe classified. For example, the type may include the type of procedurebeing performed, the phase of the procedure, whether or not theintraoperative surgical event is adverse, whether the intraoperativesurgical event is part of the planned procedure, the identity of asurgeon performing the intraoperative surgical event, a purpose of theintraoperative surgical event, a medical condition associated with theintraoperative surgical event, or any other category or classification.

Embodiments of the present disclosure may further include exporting thefirst group of frames for storage in a medical record of the patient. Asdescribed above, the particular surgical footage may depict a surgicalprocedure performed on a patient. Using the disclosed methods, the firstgroup of frames associated with the at least one interoperative surgicalevent may be associated with the patient's medical record. As usedherein, a medical record may include any form of documentation ofinformation relating to a patient's health, including diagnoses,treatment, and/or care. The medical record may be stored in a digitalformat, such as an electronic medical record (EMR). Exporting the firstgroup of frames may include transmitting or otherwise making the firstgroup of frames available for storage in the medical record or in amanner otherwise associating the first group of frames with the medicalrecord. This may include, for example, transmitting the first group offrames (or copies of the first group of frames) to an external device,such as a database. In some embodiments, the disclosed methods mayinclude associating the first group of frames with a unique patientidentifier and updating a medical record including the unique patientidentifier. The unique patient identifier may be any indicator, such asan alphanumerical string, that uniquely identifies the patient. Thealphanumeric string may anonymize the patient, which may be required forprivacy purposes. In instances where privacy may not be an issue, theunique patient identifier may include a name and/or social securitynumber of the patient.

In some exemplary embodiments, the disclosed methods may furthercomprise generating an index of the at least one intraoperative surgicalevent. As described above, an index may refer to a form of data storagethat enables retrieval of the associated video frames. Indexing mayexpedite retrieval in a manner more efficient and/or effective than ifnot indexed. The index may include a list or other itemization ofintraoperative surgical events depicted in or otherwise associated withthe first group of frames. Exporting the first group of frames mayinclude generating a compilation of the first group of frames, thecompilation including the index and being configured to enable viewingof the at least one intraoperative surgical event based on a selectionof one or more index items. For example, by selecting “incision” throughthe index, the user may be presented with a compilation of surgicalfootage depicting incisions. Various other intraoperative surgicalevents may be included on the index. In some embodiments, thecompilation may contain a series of frames of differing intraoperativeevents stored as a continuous video. For example, the user may selectmultiple intraoperative events through the index, and frames associatedwith the selected intraoperative events may be compiled into a singlevideo.

Embodiments of the present disclosure may further include generating acause effect summary. The cause-effect summary may allow a user to viewclips or images associated with a cause phase of a surgical procedureand clips or images of associated outcome phase, without having to viewintermediate clips or images. As used herein “cause” refers to triggeror action that gives rise to a particular result, phenomenon orcondition. The “outcome” refers to the phenomenon or condition that canbe attributed to the cause. In some embodiments, the outcome may be anadverse outcome. For example, the outcome may include a bleed,mesenteric emphysema, injury, conversion to unplanned open surgery (forexample, abdominal wall incision), an incision that is significantlylarger than planned, and so forth. The cause may an action, such as anerror by the surgeon, that results in or can be attributed to theadverse outcome. For example, the error may include a technical error, acommunication error, a management error, a judgment error, adecision-making error, an error related to medical equipmentutilization, or other forms of errors that may occur. The outcome mayalso include a positive or expected outcome, such as a successfuloperation, procedure, or phase.

In embodiments where a cause-effect summary is generated, the historicaldata may further include historical surgical outcome data and respectivehistorical cause data. The historical surgical outcome data may indicateportions of the historical surgical footage associated with an outcomeand the historical cause data may indicate portions of the historicalsurgical footage associated with a respective cause of the outcome. Insuch embodiments, the first group of frames may include a cause set offrames and an outcome set of frames, whereas the second group of framesmay include an intermediate set of frames.

FIG. 10 is a flowchart illustrating an exemplary process 1000 forgenerating a cause-effect summary, consistent with the disclosedembodiments. Process 1000 is provided by way of example, and a person ofordinary skill would appreciate various other processes for generating acause-effect summary consistent with this disclosure. At step 1010,process 1000 may include analyzing the particular surgical footage toidentify a surgical outcome and a respective cause of the surgicaloutcome, the identifying being based on the historical outcome data andrespective historical cause data. The analysis may be performed usingimage and/or video processing algorithms, as discussed above. In someembodiments, step 1010 may include using a machine learning modeltrained to identify surgical outcomes and respective causes of thesurgical outcomes using the historical data to analyze the particularsurgical footage. For example, the machine learning model may be trainedbased on historical data with known or predetermined surgical outcomesand respective causes. The trained model may then be used to identifysurgical outcomes and respective causes in other footage, such as theparticular surgical footage. An example of a training examples used totrain such machine learning model may include a video clip of a surgicalprocedure, together with a label indicating a surgical outcomecorresponding to the video clip, and possibly a respective cause of thesurgical outcome. Such training example may be based on the historicaldata, for example including a video clip from the historical data,including an outcome determined based on the historical data, and soforth.

At step 1020, process 1000 may include detecting, based on theanalyzing, the outcome set of frames in the particular surgical footage,the outcome set of frames being within an outcome phase of the surgicalprocedure. The outcome phase may be a timespan or portion of a surgicalprocedure that is associated with an outcome as described above. At step1030, process 1000 may include detecting, based on the analyzing, acause set of frames in the particular surgical footage, the cause set offrames being within a cause phase of the surgical procedure remote intime from the outcome phase. In some embodiments, the outcome phase mayinclude a surgical phase in which the outcome is observable, and theoutcome set of frames may be a subset of frames in the outcome phase.The cause phase may be a timespan or portion of the surgical procedurethat is associated with a cause of the outcome in the outcome phase. Insome embodiments, the cause phase may include a surgical phase in whichthe cause occurred, and the cause set of frames may be a subset of theframes in the cause phase. The intermediate set of frames may be withinan intermediate phase interposed between the cause set of frames and theoutcome set of frames. At step 1040, process 1000 may include generatinga cause-effect summary of the surgical footage, wherein the cause-effectsummary includes the cause set of frames and the outcome set of framesand omits the intermediate set of frames. In some embodiments, thecause-effect summary may be similar to the aggregate of the first groupof frames, as described above. Accordingly, the cause-effect summary mayinclude a compilation of video clips associated with the cause set offrames and outcome set of frames. The aggregate of the first group offrames presented to the user, as described above, may include the causeeffect summary.

FIG. 11 is a flowchart illustrating an example process 1100 forgenerating surgical summary footage, consistent with the disclosedembodiments. Process 1100 may be performed by a processing device. Insome embodiments, a non-transitory computer readable medium may containinstructions that when executed by a processor cause the processor toperform process 1100. At step 1110, process 1100 may include accessingparticular surgical footage containing a first group of framesassociated with at least one intraoperative surgical event and a secondgroup of frames not associated with surgical activity. As discussed infurther detail above, the first group of frames may be associated withmultiple intraoperative surgical events and may not necessarily beconsecutive frames. Further, in some embodiments, the first group offrames may include a cause set of frames and an outcome set of frames,and the second group of frames may include an intermediate set offrames, as discussed above with respect to process 1000.

At step 1120, process 1100 may include accessing historical data basedon historical surgical footage of prior surgical procedures, wherein thehistorical data includes information that distinguishes portions ofsurgical footage into frames associated with intraoperative surgicalevents and frames not associated with surgical activity. In someembodiments, the information that distinguishes portions of thehistorical surgical footage into frames associated with anintraoperative surgical event may include an indicator of at least oneof a presence or a movement of a surgical tool and/or an anatomicalfeature. At step 1130, process 1100 may include distinguishing in theparticular surgical footage the first group of frames from the secondgroup of frames based on the information of the historical data.

At step 1140, process 1100 may include, upon request of a user,presenting to the user an aggregate of the first group of frames of theparticular surgical footage, while omitting presentation to the user ofthe second group of frames. The request of the user may be received froma computing device which may include a user interface enabling the userto make the request. In some embodiments, the user may further requestframes associated with a particular type or category of intraoperativeevents. Based on the steps described in process 1100, the user may bepresented a summary including frames associated with intraoperativeevents and omitting frames not associated with surgical activity. Thesummary may be used, for example, by a surgeon as a training video thataggregates the intraoperative surgical events, while omitting much ofthe other irrelevant footage.

When preparing for a surgical procedure, it may be beneficial for asurgeon to review video footage of several surgical procedures havingsimilar surgical events. Conventional approaches may not allow a surgeonto easily access video footage of surgical procedures having similarsurgical events. Further, even if the footage is accessed, it may be tootime consuming to watch the entire video or to find relevant portions ofthe videos. Therefore, there is a need for unconventional approachesthat efficiently and effectively enable a surgeon to view a videocompiling footage of surgical events from surgeries performed ondifferent patients.

Aspects of this disclosure may relate to surgical preparation, includingmethods, systems, devices, and computer readable media. In particular, acompilation video of differing events in surgeries performed ondifferent patients may be presented to a surgeon or other user. Thecompilation may include excerpts of surgical video of differingintraoperative events from similar surgical procedures, which may beautomatically aggregated in a composite form. A surgeon may be enabledto input case-specific information, to retrieve the compilation of videosegments selected from similar surgeries on different patients. Thecompilation may include one intraoperative event from one surgery andother different intraoperative events from one or more second surgeries.For example, different complications that occur when operating ondifferent patients may all be included in one compilation video. Insituations where videos of multiple surgical procedures contain the sameevent with a shared characteristic (e.g., a similar technique employed),the system may omit footage from one or more surgical procedures toavoid redundancy.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In a broadest sense, the method is not limited toparticular physical and/or electronic instrumentalities, but rather maybe accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve accessing arepository of a plurality of sets of surgical video footage. As usedherein, a repository may refer to any storage location or set of storagelocations where video footage may be stored electronically. For example,the repository may include a memory device, such as a hard drive and/orflash drive. In some embodiments, the repository may be a networklocation such as a server, a cloud storage location, a shared networkdrive, or any other form of storage accessible over a network. Therepository may include a database of surgical video footage captured atvarious times and/or locations. In some embodiments, the repository maystore additional data besides the surgical video footage.

As described above, surgical video footage may refer to any video, groupof video frames, or video footage including representations of asurgical procedure. For example, the surgical footage may include one ormore video frames captured during a surgical operation. A set ofsurgical video footage may refer to a grouping of one or more surgicalvideos or surgical video clips. The video footage may be stored in thesame location or may be selected from a plurality of storage locations.Although not necessarily so, videos within a set may be related in someway. For example, video footage within a set may include videos,recorded by the same capture device, recorded at the same facility,recorded at the same time or within the same timeframe, depictingsurgical procedures performed on the same patient or group of patients,depicting the same or similar surgical procedures, depicting surgicalprocedures sharing a common characteristic (such as similar complexitylevel, including similar events, including usages of similar techniques,including usages of similar medical instruments, etc.), or sharing anyother properties or characteristics.

The plurality of sets of surgical video footage may reflect a pluralityof surgical procedures performed on differing patients. For example, anumber of different individuals who underwent the same or similarsurgical procedure, or who underwent surgical procedures where a similartechnique was employed may be included within a common set or aplurality of sets. Alternatively or in addition, one or more sets mayinclude surgical footage captured from a single patient but at differenttimes. The plurality of surgical procedures may be of the same type, forexample, all including appendectomies, or may be of different types. Insome embodiments, the plurality of surgical procedures may share commoncharacteristics, such as the same or similar phases or intraoperativeevents.

The plurality of sets of surgical video footage may further includeintraoperative surgical events, surgical outcomes, patientcharacteristics, surgeon characteristics, and intraoperative surgicalevent characteristics. Examples of such events, outcomes, andcharacteristics are described throughout the present disclosure. Asurgical outcome may include outcomes of the surgical procedure as awhole (e.g., whether the patient recovered or recovered fully, whetherpatient was readmitted after discharge, whether the surgery wassuccessful), or outcomes of individual phases or events within thesurgical procedure (e.g., whether a complication occurred or whether atechnique was successful).

Some aspects of the present disclosure may involve enabling a surgeonpreparing for a contemplated surgical procedure to input case-specificinformation corresponding to the contemplated surgical procedure. Acontemplated surgical procedure may include any surgical procedure thathas not already been performed. In some embodiments, the surgicalprocedure may be a planned surgical procedure that the surgeon intendsto perform on a patient. In other embodiments the contemplated surgicalprocedure may be a hypothetical procedure and may not necessarily beassociated with a specific patient. In some embodiments, thecontemplated surgical procedure may be experimental and may not be inwidespread practice. The case-specific information may include anycharacteristics or properties of the contemplated surgical procedure orof a contemplated or hypothetical patient. For example, thecase-specific information may include, but is not limited to,characteristics of the patient the procedure will be performed on,characteristics of the surgeon performing the procedure, characteristicsof other healthcare professionals involved in the procedure, the type ofprocedure being performed, unique details or aspects of the procedure,the type of equipment or tools involved, types of technology involved,complicating factors of the procedure, a location of the procedure, thetype of medical condition being treated or certain aspects thereof, asurgical outcome, an intraoperative event outcome, or any otherinformation that may define or describe the contemplated surgicalprocedure. For example, the case-specific information may include apatient's age, weight, medical condition, vital signs, other physicalcharacteristics, past medical history, family medical history, or anyother type of patient-related information that might have some direct orindirect bearing on a potential outcome. The case-specific informationmay also include an indicator of the performing surgeon's skill level, asurgical technique employed, a complication encountered, or any otherinformation about the surgeon, the procedure, the tools used, or thefacility.

The case-specific information may be input in various ways. In someembodiments, the surgeon may input the case-specific information througha graphical user interface. The user interface may include one or moretext fields, prompts, drop-down lists, checkboxes or other fields ormechanisms for inputting the information. In some embodiments, thegraphical user interface may be associated with the computing device orprocessor performing the disclosed methods. In other embodiments, thegraphical user interface may be associated with an external computingdevice, such as a mobile phone, a tablet, a laptop, a desktop computer,a computer terminal, a wearable device (including smart watches, smartglasses, smart jewelry, head-mounted displays, etc.), or any otherelectronic device capable of receiving a user input. In someembodiments, the case-specific information may be input at an earliertime or over a period of time (e.g., several days, several months,several years, or longer). Some or all of the case-specific informationmay be extracted from a hospital or other medical facility database, anelectronic medical record, or any other location that may store patientdata and/or other medical data. In some embodiments, the case-specificinformation corresponding to the contemplated surgical procedure may bereceived from an external device. For example, the case-specificinformation may be retrieved or otherwise received from an externalcomputing device, a server, a cloud-computing service, a network device,or any other device external to the system performing the disclosedmethods. In one example, at least part of the case-specific informationcorresponding to the contemplated surgical procedure may be receivedfrom an Electronic Health Record (EMR) or from a system handling the EMR(for example, an EMR of a particular patient the procedure will beperformed on, an EMR associated with the contemplated surgicalprocedure, etc.), from a scheduling system, from electronic recordscorresponding to a medical professional associated with the contemplatedsurgical procedure or from a system handling the electronic record, andso forth.

In some exemplary embodiments, the case-specific information may includea characteristic of a patient associated with the contemplatedprocedure. For example, as mentioned earlier, the case-specificinformation may include characteristics of a contemplated patient.Patient characteristics may include, but are not limited to, a patient'sgender, age, weight, height, physical fitness, heart rate, bloodpressure, temperature, medical condition or disease, medical history,previous treatments, or any other relevant characteristic. Otherexemplary patient characteristics are described throughout the presentdisclosure. In some embodiments, a characteristic of the patient may beentered directly by the surgeon. For example, a patient characteristicmay be entered through a graphical user interface, as described above.In other embodiments, the characteristic of the patient may be retrievedfrom a database or other electronic storage location. In someembodiments, the characteristic of the patient may be received from amedical record of the patient. For example, a patient characteristic maybe retrieved from the medical record or other information source basedon an identifier or other information input by the surgeon. For example,the surgeon may enter a patient identifier and the medical record of thepatient and/or the patient characteristic may be retrieved using thepatient identifier. As describe herein, the patient identifier may beanonymous (e.g., an alphanumeric code or machine readable code) or itmay identify the patient in a discernable way (e.g., patient name orsocial security number). In some examples, the case-specific informationmay include characteristics of two or more patients associated with thecontemplated procedure (for example, for contemplated surgicalprocedures that involves two or more patients, such as transplants)

In accordance with the present disclosure, the case-specific informationmay include information relating to a surgical tool. The surgical toolmay be any device or instrument used as part of a surgery. Someexemplary surgical tools are described throughout the presentdisclosure. In some embodiments, the information relating to thesurgical tool may include at least one of a tool type or a tool model. Atool type may refer to any classification of the tool. For example, thetool type may refer to the kind of instrument being used (e.g.,“scalpel,” “scissors,” “forceps,” “retractor,” or other kinds ofinstruments). Tool type may include various other classifications, suchas whether the tool is electronic, whether the tool is used for aminimally invasive surgery, the materials the tool is constructed of, asize of the tool, or any other distinguishing properties. The tool modelmay refer to the specific make and/or manufacturer of the instrument(e.g., “15921 Halsted Mosquito Forceps”).

Embodiments of the present disclosure may further include comparing thecase-specific information with data associated with the plurality ofsets of surgical video footage to identify a group of intraoperativeevents likely to be encountered during the contemplated surgicalprocedure. Data associated with the plurality of sets of surgical videosmay include any stored information regarding the surgical video footage.The data may include information identifying intraoperative surgicalevents, surgical phases, or surgical event characteristics depicted inor associated with the surgical video footage. The data may includeother information such as patient or surgeon characteristics, propertiesof the video (e.g., capture date, file size, information about thecapture device, capture location, etc.) or any other informationpertaining to the surgical video footage. The data may be stored as tagsor other data within the video files. In other embodiments, the data maybe stored in a separate file. In some embodiments the surgical videofootage may be indexed to associate the data with the video footage.Accordingly, the data may be stored in a data structure, such as datastructure 600, described above. In one example, comparing thecase-specific information with data associated one or more surgicalvideo footage (for example, with the plurality of sets of surgical videofootage) may include calculating one or more similarity measures betweenthe case-specific information and the data associated one or moresurgical video footage, for example using one or more similarityfunctions. Further, in one example, the calculated similarity measuresmay be compared with selected threshold to determine if an event thatoccurred in the one or more surgical video footage is likely to occur inthe contemplated surgical procedure, for example using a k-NearestNeighbors algorithm to predict that events commonly occurring the k mostsimilar surgical video footage are likely to be encountered during thecontemplated surgical procedure. In some examples, a machine learningmodel may be trained using training examples to identify intraoperativeevents likely to be encountered during specific surgical procedures frominformation related to the specific surgical procedures, and the trainedmachine learning model may be used to analyze the case-specificinformation of the contemplated surgical procedure and identify thegroup of intraoperative events likely to be encountered during thecontemplated surgical procedure. An example of such training example mayinclude information related to a particular surgical procedure, togetherwith a label indicating intraoperative events likely to be encounteredduring the particular surgical procedure.

The group of intraoperative events likely to be encountered during thecontemplated surgical procedure may be determined based on the data. Forexample, the case-specific information may be compared to the dataassociated with the plurality of sets of surgical video footage. Thismay include comparing characteristics of the contemplated surgicalprocedure (as represented in the case-specific information) to identifysurgical video footage associated with surgical procedures having thesame or similar characteristics. For example, if the case-specificinformation includes a medical condition of a patient associated withthe contemplated procedure, sets of surgical video footage associatedwith surgical procedures on patients with the same or similar medicalconditions may be identified. By way of another example, a surgeonpreparing to perform a catheterization on a 73 year old male withdiabetes, high cholesterol, high blood pressure, and a family history ofheart disease, may enter that case-specific information in order to drawvideo footage for review of patients sharing similar characteristics (orpatients predicted to present similarly to the specific patient). Thegroup of intraoperative events likely to be encountered during thecontemplated surgical procedure may include intraoperative surgicalevents that were encountered during the surgical procedures associatedwith the identified surgical video footage. In some embodiments,multiple factors may be considered in identifying the surgical videofootage and/or the group of intraoperative events likely to beencountered.

Whether an intraoperative event is considered likely to be encounteredduring the contemplated surgical procedure may depend on how frequentlythe intraoperative event occurs in surgical procedures similar to thecontemplated surgical procedure. For example, the intraoperative eventmay be identified based on the number of times it occurs in similarprocedures, the percentage of times it occurs in similar procedures, orother statistical information based on the plurality of sets of surgicalvideo footage. In some embodiments, intraoperative events may beidentified based on comparing the likelihood to a threshold. Forexample, an intraoperative event may be identified if it occurs in morethan 50% of similar surgical procedures, or any other percentage. Insome embodiments, the group of intraoperative events may include tiersof intraoperative events based on their likelihood of occurrence. Forexample, group may include a tier of intraoperative events with a highlikelihood of occurrence and one or more tiers of intraoperative eventswith a lower likelihood of occurrence.

In accordance with some embodiments of the present disclosure, machinelearning or other artificial intelligence techniques may be used toidentify the group of intraoperative events. Accordingly, comparing thecase-specific information with data associated with the plurality ofsets of surgical video footage may include using an artificial neuralnetwork to identify the group of intraoperative events likely to beencountered during the contemplated surgical procedure. In one example,the artificial neural network may be configured manually, may begenerated from a combination of two or more other artificial neuralnetworks, and so forth. In one example, the artificial neural networkmay be fed training data correlating various case-specific informationwith intraoperative events likely to be encountered. In someembodiments, the training data may include one or more sets of surgicalvideo footage included in the repository and data associated with thesurgical footage. The training data may also include non-video relateddata, such as patient characteristics or past medical history. Using anartificial neural network, a trained model may be generated based on thetraining data. Accordingly, using the artificial neural network mayinclude providing the case-specific information to the artificial neuralnetwork as an input. As an output of the model, the group ofintraoperative events likely to be encountered during the contemplatedsurgical procedure may be identified. Various other machine learningalgorithms may be used, including a logistic regression, a linearregression, a regression, a random forest, a K-Nearest Neighbor (KNN)model (for example as described above), a K-Means model, a decisiontree, a cox proportional hazards regression model, a Naïve Bayes model,a Support Vector Machines (SVM) model, a gradient boosting algorithm, orany other form of machine learning model or algorithm.

Some aspects of the present disclosure may further include using thecase-specific information and the identified group of intraoperativeevents likely to be encountered to identify specific frames in specificsets of the plurality of sets of surgical video footage corresponding tothe identified group of intraoperative events. The specific frames inspecific sets of the plurality of sets of surgical video footage may belocations in the video footage where the intraoperative events occur.For example, if the group of intraoperative events includes acomplication, the specific frames may include video footage depictingthe complication or otherwise associated with the complication. In someembodiments, the specific frames may include some surgical video footagebefore or after occurrence of the intraoperative event, for example, toprovide context for the intraoperative event. Further, the specificframes may not necessarily be consecutive. For example, if theintraoperative event is an adverse event or outcome, the specific framesmay include frames corresponding to the adverse outcome and a cause ofthe adverse outcome, which may not be consecutive. The specific framesmay be identified based on frame numbers (e.g., a frame number, abeginning frame number and an ending frame number, a beginning framenumber and a number of subsequent frames, etc.), based on timeinformation (e.g., a start time and stop time, a duration, etc.), or anyother manner for identifying specific frames of video footage.

In some embodiments, the specific frames may be identified based onindexing of the plurality of surgical video footage. For example, asdescribed above, video footage may be indexed to correlate footagelocations to phase tags, event tags, and or event characteristics.Accordingly, identifying the specific frames in specific sets of theplurality of sets of surgical video footage may include performing alookup or search for the intraoperative events using a data structure,such as data structure 600 as described in relation to FIG. 6 .

In accordance with the present disclosure, the identified specificframes may include frames from the plurality of surgical proceduresperformed on differing patients. Accordingly, the identified specificframes may form a compilation of footage associated with intraoperativeevents from surgical procedures performed on different patients, whichmay be used for surgical preparation. For example, the best video clipexamples (in terms of video quality, clarity, representativeness,compatibility with the contemplated surgical procedure, etc.) may bechosen from differing procedures performed on differing patients, andassociated with each other so that a preparing surgeon can view the bestof a group of video clips, for example without having to separatelyreview video of each case, one by one.

Embodiments of the present disclosure may further include omittingportions of the identified specific frames, for example, to avoidredundancy, to shorten the resulting compilation, to remove lessrelevant or less informative portions, and so forth. Accordingly, someembodiments may include determining that a first set and a second set ofvideo footage from differing patients contain frames associated withintraoperative events sharing a common characteristic. The first set andsecond set of video footage may comprise frames of the identifiedspecific frames corresponding to the identified group of intraoperativeevents. The common characteristic may be any characteristic of theintraoperative events that is relevant to determining whether framesfrom the first set and the second set should both be included. Thecommon characteristic may be used to determine whether the first set andthe second set are redundant. For example, the intraoperative event maybe a complication that occurs during the surgical procedure and thecommon characteristic may be a type of complication. If thecomplications in first and seconds sets of frames are of the same type,it may not be efficient or beneficial for a surgeon preparing forsurgery to view both the first set and second set of frames. Thus, onlyone set may be chosen for presentation to the surgeon, with the otherset being omitted. In some embodiments of the present disclosure, thecommon characteristic may include a characteristic of the differingpatients. For example, the common characteristic may include a patient'sage, weight, height, or other demographics, may include patientcondition, and so forth. Various other patient characteristics describedthroughout the present disclosure may also be shared. In otherembodiments, the common characteristic may include an intraoperativesurgical event characteristic of the contemplated surgical procedure.The intraoperative surgical event characteristic may include any traitor property of the intraoperative event. For example, an adverse outcomeof the surgical event, a surgical technique, a surgeon skill level, anidentity of a specific surgeon, a physiological response, duration ofthe event, or any other characteristic or property associated with theevent.

According to various exemplary embodiments of the present disclosure,determining that a first set and a second set of video footage fromdiffering patients contain frames associated with intraoperative eventssharing a common characteristic may include using an implementation of amachine learning model to identify the common characteristic. In oneexample, a machine learning model may be trained using training examplesto identify frames of video footage having particular characteristics,and the trained machine learning model may be used to analyze the firstset and the second set of video footage from differing patients toidentify the frames associated with intraoperative events sharing acommon characteristic. An example of such training example may include avideo clip together with a label indicating particular characteristicsof particular frames of the video clip. Various machine learning modelsare described above and may include a logistic regression model, alinear regression model, a regression model, a random forest model, aK-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a coxproportional hazards regression model, a Naïve Bayes model, a SupportVector Machines (SVM) model, a gradient boosting algorithm, a deeplearning model, or any other form of machine learning model oralgorithm. Some embodiments of the present disclosure may furtherinclude using example video footage to train the machine learning modelto determine whether two sets of video footage share the commoncharacteristic, and wherein implementing the machine learning modelincludes implementing the trained machine learning model. In oneexample, the example video footage may be training footage, which mayinclude pairs of sets of video footage known to share the commoncharacteristic. The trained machine learning model may be configured todetermine whether two sets of video footage share the commoncharacteristic.

The disclosed embodiments may further include omitting an inclusion ofthe second set from a compilation to be presented to the surgeon andincluding the first set in the compilation to be presented to thesurgeon. As used herein, a compilation may include a series of framesthat may be presented for continuous and/or consecutive playback. Insome embodiment, the compilation may be stored as a separate video file.In other embodiments, the compilation may be stored as instructions topresent the series of frames from their respective surgical videofootage, for example, stored in the repository. The compilation mayinclude additional frames besides those included in the first set,including other frames from the identified specific frames.

Some aspects of the present disclosure may further include enabling thesurgeon to view a presentation including the compilation containingframes from the differing surgical procedures performed on differingpatients. The presentation may be any form of visual display includingthe compilation of frames. In some embodiments the presentation may be acompilation video. The presentation may include other elements, such asmenus, controls, indices, timelines, or other content in addition to thecompilation. In some embodiments, enabling the surgeon to view thepresentation may include outputting data for displaying the presentationusing a display device, such as a screen (e.g., an OLED, QLED LCD,plasma, CRT, DLPT, electronic paper, or similar display technology), alight projector (e.g., a movie projector, a slide projector), a 3Ddisplay, smart glasses, or any other visual presentation mechanism, withor without audio presentation. In other embodiments, enabling thesurgeon to view the presentation may include storing the presentation ina location that is accessible by one or more other computing devices.Such storage locations may include a local storage (such as a hard driveof flash memory), a network location (such as a server or database), acloud computing platform, or any other accessible storage location.Accordingly, the presentation may be accessed from an external device tobe displayed on the external device. In some embodiments, outputting thevideo may include transmitting the video to an external device. Forexample, enabling the surgeon to view the presentation may includetransmitting the presentation through a network to a user device orother external device for playback on the external device.

The presentation may stitch together disparate clips from differingprocedures, presenting them to the surgeon in the chronological order inwhich they might occur during surgery. The clips may be presented toplay continuously, or may be presented in a manner requiring the surgeonto affirmatively act in order for a succeeding clip to play. In someinstances where it may be beneficial for the surgeon to view multiplealternative techniques or to view differing responses to adverse events,multiple alternative clips from differing surgical procedures may bepresented sequentially.

Some embodiments of the present disclosure may further include enablinga display of a common surgical timeline including one or morechronological markers corresponding to one or more of the identifiedspecific frames along the presentation. For example, the common surgicaltimeline may be overlaid on the presentation, as discussed above. Anexample surgical timeline 420 including chronological markers is shownin FIG. 4 . The chronological markers may correspond to markers 432,434, and/or 436. Accordingly, the chronological markers may correspondto a surgical phase, an intraoperative surgical event, a decision makingjunction, or other notable occurrences the identified specific framesalong the presentation. The markers may be represented as shapes, icons,or other graphical representations along the timeline, as described infurther detail above. The timeline may be presented together with framesassociated with a surgery performed on a single patient, or may bepresented together with a compilation of video clips from surgeriesperformed on a plurality of patients.

In accordance with some embodiments of the present disclosure, enablingthe surgeon to view the presentation may include sequentially displayingdiscrete sets of video footage of the differing surgical proceduresperformed on differing patients. Each discrete set of video footage maycorrespond to a different surgical procedure performed on a differentpatient. In some embodiments, sequentially displaying the discrete setsof video footage may appear to the surgeon or another user as acontinuous video. In other embodiments playback may stop or pausebetween the discrete sets of video footage. The surgeon or other usermay manually start the next set of video footage in the sequence.

In accordance with some embodiments of the present disclosure, thepresentation may include a display of a simulated surgical procedurebased on the identified group of intraoperative events likely to beencountered and/or the identified specific frames in specific sets ofthe plurality of sets of surgical video footage corresponding to theidentified group of intraoperative events. For example, a machinelearning algorithm (such as a Generative Adversarial Network) may beused to train a machine learning model (such as an artificial neuralnetwork, a deep learning model, a convolutional neural network, etc.)using training examples to generate simulations of surgical proceduresbased on groups of intraoperative events and/or frames of surgical videofootage, and the trained machine learning model may be used to analyzethe identified group of intraoperative events likely to be encounteredand/or the identified specific frames in specific sets of the pluralityof sets of surgical video footage corresponding to the identified groupof intraoperative events and generate the simulated surgical procedure.

In some embodiments, sequentially displaying discrete sets of videofootage may include displaying an index of the discrete sets of videofootage enabling the surgeon or other user to select one or more of thediscrete sets of video footage. The index may be a text-based index, forexample, listing intraoperative events, surgical phases, or otherindicators of the different discrete sets of video footage. In otherembodiments, the index may be a graphical display, such as a timeline asdescribed above, or a combination of graphical and textual information.For example, the index may include a timeline parsing the discrete setsinto corresponding surgical phases and textual phase indicators. In suchembodiments, the discrete sets may correspond to different surgicalphases of the surgical procedure. The discrete sets may be displayedusing different colors, with different shading, with bounding boxes orseparators, or other visual indicators to distinguish the discrete sets.The textual phase indicators may describe or otherwise identify thecorresponding surgical phase. The textual phase indicators may bedisplayed within the timeline, above the timeline, below the timeline orin any location such that they identify the discrete sets. In someembodiments, the timeline may be displayed in a list format and thetextual phase indicators may be included within the list.

In accordance with the present disclosure, the timeline may include anintraoperative surgical event marker corresponding to an intraoperativesurgical event. The intraoperative surgical event marker may correspondto an intraoperative surgical event associated with a location in thesurgical video footage. The surgeon may be enabled to click on theintraoperative surgical event marker to display at least one framedepicting the corresponding intraoperative surgical event. For example,clicking on the intraoperative surgical event may cause a display of thecompilation video to skip to a location associated with the selectedmarker. In some embodiments, the surgeon may be able to view additionalinformation about the event or occurrence associated with the marker,which may include information summarizing aspects of the procedure orinformation derived from past surgical procedures, as described ingreater detail above. Any of the features or functionality describedabove with respect to timeline overlay on surgical video may also applyto the compilation videos described herein.

Embodiments of the present disclosure may further include training amachine learning model to generate an index of the repository based onthe intraoperative surgical events, the surgical outcomes, the patientcharacteristics, the surgeon characteristics, and the intraoperativesurgical event characteristics and generating the index of therepository. Comparing the case-specific information with data associatedwith the plurality of sets may include searching the index. The variousmachine learning models described above, including a logistic regressionmodel, a linear regression model, a regression model, a random forestmodel, a K-Nearest Neighbor (KNN) model, a K-Means model, a decisiontree, a cox proportional hazards regression model, a Naïve Bayes model,a Support Vector Machines (SVM) model, a gradient boosting algorithm, adeep learning model, or any other form of machine learning model oralgorithm may be used. A training data set of surgical video footagewith known intraoperative surgical events, surgical outcomes, patientcharacteristics, surgeon characteristics, and intraoperative surgicalevent characteristics may be used to train the model. The trained modelmay be configured to determine intraoperative surgical events, surgicaloutcomes, patient characteristics, surgeon characteristics, andintraoperative surgical event characteristics based on additionalsurgical video footage not included in the training set. When applied tosurgical video footage in the repository, the video footage may betagged based on the identified properties. For example, the videofootage may be associated with a footage location, phase tag, eventlocation, and/or event tag as described above with respect to videoindexing. Accordingly, the repository may be stored as a data structure,such as data structure 600, described above.

FIG. 12 is a flowchart illustrating an example process 1200 for surgicalpreparation, consistent with the disclosed embodiments. Process 1200 maybe performed by a processing device, such as one or more collocated ordispersed processors as described herein. In some embodiments, anon-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform process1200. Process 1200 is not necessarily limited to the steps shown in FIG.1200 and any steps or processes of the various embodiments describedthroughout the present disclosure may also be included in process 1200.At step 1210, process 1200 may include accessing a repository of aplurality of sets of surgical video footage reflecting a plurality ofsurgical procedures performed on differing patients. The plurality ofsets of surgical video footage may include intraoperative surgicalevents, surgical outcomes, patient characteristics, surgeoncharacteristics, and intraoperative surgical event characteristics. Insome embodiments, the repository may be indexed, for example usingprocess 800, to facilitate retrieval and identification of the pluralityof sets of surgical video footage.

At step 1220, process 1200 may include enabling a surgeon preparing fora contemplated surgical procedure to input case-specific informationcorresponding to the contemplated surgical procedure. As describedabove, the contemplated surgical procedure may be a planned procedure, ahypothetical procedure, an experimental procedure, or another procedurethat has not yet occurred. The case-specific information may be manuallyinput by the surgeon, for example through a user interface. In someembodiments, some or all of the case-specific information may bereceived from a medical record of the patient. The case-specificinformation may include a characteristic of a patient associated withthe contemplated procedure, information includes information relating toa surgical tool (e.g., a tool type, a tool model, a tool manufacturer,etc.), or any other information that may be used to identify relevantsurgical video footage.

At step 1230, process 1200 may include comparing the case-specificinformation with data associated with the plurality of sets of surgicalvideo footage to identify a group of intraoperative events likely to beencountered during the contemplated surgical procedure. The group ofintraoperative events likely to be encountered may be determined, forexample, based on machine learning analyses performed on historicalvideo footage, historical data other than video data, or any other formof data from which a prediction may be derived. At step 1240, process1200 may include using the case-specific information and the identifiedgroup of intraoperative events likely to be encountered to identifyspecific frames in specific sets of the plurality of sets of surgicalvideo footage corresponding to the identified group of intraoperativeevents. The identified specific frames may include frames from theplurality of surgical procedures performed on differing patients, asdescribed earlier.

At step 1250, process 1200 may include determining that a first set anda second set of video footage from differing patients contain framesassociated with intraoperative events sharing a common characteristic,as described earlier. At step 1260, process 1200 may include omitting aninclusion of the second set from a compilation to be presented to thesurgeon and including the first set in the compilation to be presentedto the surgeon, as described earlier.

At step 1270, process 1200 may include enabling the surgeon to view apresentation including the compilation containing frames from thediffering surgical procedures performed on differing patients. Asdescribed above, enabling the surgeon to view the presentation mayinclude outputting data to enable displaying the presentation on ascreen or other display device, storing the presentation in a locationaccessible to another computing device, transmitting the presentation,or any other process or method that may cause the enable thepresentation and/or compilation to be viewed.

When preparing for a surgical procedure, it may be beneficial for asurgeon to review video footage of past surgical procedures. However, insome instances, only particularly complex portions of the surgicalprocedures may be relevant to the surgeon. Using conventionalapproaches, it may be too difficult and time consuming for a surgeon toidentify portions of a surgical video based on the complexity of theprocedure. Therefore, there is a need for unconventional approaches forefficiently and effectively analyzing complexity of surgical footage andenabling a surgeon to quickly review relevant portions of a surgicalvideo.

Aspects of this disclosure may relate to surgical preparation, includingmethods, systems, devices, and computer readable media. In particular,when preparing for a surgical procedure, surgeons may wish to viewportions of surgical videos that have particular complexity levels. Forexample, within a generally routine surgical video, a highly skilledsurgeon may wish to view only a single event that was unusually complex.Finding the appropriate video and the appropriate location in the video,however, can be time consuming for the surgeon. Accordingly, in someembodiments, methods and systems for analyzing complexity of surgicalfootage are provided. For example, the process of viewing surgical videoclips based on complexity may be accelerated by automatically taggingportions of surgical video with a complexity score, thereby permitting asurgeon to quickly find the frames of interest based on complexity.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In a broadest sense, the method is not limited toparticular physical and/or electronic instrumentalities, but rather maybe accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method may involve analyzingframes of the surgical footage to identify in a first set of frames ananatomical structure. As described above, surgical footage may refer toany video, group of video frames, or video footage includingrepresentations of a surgical procedure. For example, the surgicalfootage may include one or more video frames captured during a surgicaloperation. The first set of frames may be a grouping of one or moreframes included within the surgical footage. In some embodiments, thefirst set of frames may be consecutive frames, however, this is notnecessarily true. For example, the first set of frames may include aplurality of groups of consecutive frames.

As discussed above, an anatomical structure may be any particular partof a living organism, including, for example organs, tissues, ducts,arteries, cells, or other anatomical parts. The first set of frames maybe analyzed to identify the anatomical structure using varioustechniques, for example as described above. In some embodiments, theframes of the surgical footage may be analyzed using object detectionalgorithms, as described above. For example, the object detectionalgorithms may be detected objects based on one or more of appearance,image features, templates, and so forth. In some embodiments,identifying the anatomical structure in a first set of frames includesusing a machine learning model trained to detect anatomical structures,for example as described above. For example, images and/or videos alongwith identifications of anatomical structures known to be depicted inthe images and/or videos may be input into a machine learning model astraining data. As a result, the trained model may be used to analyze thesurgical footage to identify in the first set of frames, an anatomicalstructure. For example, an artificial neural network configured toidentify anatomical structures in images and/or videos may be used toanalyze the surgical footage to identify in the first set of frames ananatomical structure. Various other machine learning algorithms may beused, including a logistic regression, a linear regression, aregression, a random forest, a K-Nearest Neighbor (KNN) model, a K-Meansmodel, a decision tree, a cox proportional hazards regression model, aNaïve Bayes model, a Support Vector Machines (SVM) model, a gradientboosting algorithm, a deep learning model, or any other form of machinelearning model or algorithm.

Some aspects of the present disclosure may further include accessingfirst historical data, the first historical data being based on ananalysis of first frame data captured from a first group of priorsurgical procedures. Generally, frame data may include any image orvideo data depicting surgical procedures as described herein. The firsthistorical data and/or the first frame data may be stored on one or morestorage locations. Accordingly, accessing the first historical data mayinclude retrieving the historical data from a storage location. In otherembodiments, accessing the first historical data may include receivingthe first historical data and/or the first frame data, for example, froman image capture device or a computing device. Consistent withembodiments of the present disclosure, accessing the first historicaldata may include retrieving or receiving the first frame data andanalyzing the first frame data to identify the first historical data.

Historical data may be any information pertaining to prior surgicalprocedures. Some non-limiting examples of such historical data aredescribed above. In some embodiments, the first historical data mayinclude complexity information associated with the first group of priorsurgical procedures. The complexity information may include any dataindicating a complexity level of the surgery, as discussed furtherbelow. The first historical data may include any other informationpertaining to the first group of surgical procedures that may be gleanedfrom the first frame data. For example, the first frame data may includeor indicate information associated with the prior surgical procedures,including anatomical structures involved, medical tools used, types ofsurgical procedures performed, intraoperative events (including adverseevents) occurring in the procedures, medical conditions exhibited by thepatient, patient characteristics, surgeon characteristics, skill levelsof surgeons or other healthcare professionals involved, timinginformation (e.g., duration of interactions between medical tools andanatomical structures, duration of a surgical phase or intraoperativeevent, time between appearance of a medical tool and a first interactionbetween the medical tool and an anatomical structure, or other relevantduration or timing information), a condition of an anatomical structure,a number of surgeons or other healthcare professionals involved, or anyother information associated with the prior surgical procedures.

In embodiments where the first historical data includes complexityinformation, such information may be indicative of or associated withthe complexity of a surgical procedure or a portion thereof. Forexample, the first historical data may include an indication of astatistical relation between a particular anatomical structure and aparticular surgical complexity level. The statistical relation may beany information that may indicate some correlation between theparticular surgical complexity level and the particular anatomicalstructure. For example, when a particular vessel is exposed in asurgical procedure, a particular portion of an organ is lacerated, or aparticular amount of blood is detected, such events may statisticallycorrelate to a surgical complexity level. Similarly, detection of a highvolume of fat or a poor condition of an organ may also correlate to acomplexity level. These are just examples, any condition or event thatcorrelates to surgical complexity may serve as an indication of surgicalcomplexity

In some embodiments, the first historical data may be identified fromthe first frame data using one or more image or video analysisalgorithms, including object detection algorithms and/or motiondetection algorithms. In other embodiments, the first historical datamay be identified from the first frame data using a machine learningmodel trained to identify historical data based on frame data. Forexample, a machine learning model may be trained using training examplesto identify historical data (as described above) from frame data, andthe trained machine learning model may be used to analyze the firstframe data to determine the first historical data. An example of suchtraining example may include an image and/or a video depicting asurgical procedure or a portion of a surgical procedure, together with alabel indicating the complexity level of the surgical procedure or ofthe portion of a surgical procedure. For example, such label may begenerated manually, may be generated by a different process, may be readfrom memory, and so forth.

Embodiments of the present disclosure may involve analyzing the firstset of frames using the first historical data and using the identifiedanatomical structure, to determine a first surgical complexity levelassociated with the first set of frames. As used herein, a complexitylevel may be a value or other classifier indicating a relativecomplexity of a surgical procedure or portion of a surgical procedure.For example, the complexity may be based on a difficulty of the surgicalprocedure relative to other surgical procedures. The difficulty may bebased on the surgeon skill level required to perform one or moretechniques involved in the surgical procedure, a likelihood ofoccurrence of an adverse event (such as tear, a bleed, an injury, orother adverse events), a success rate of the surgical procedure, or anyother indicator of difficulty of the procedure. Surgical procedures withhigher relative difficulty levels may be associated with highercomplexity levels.

As another illustrative example, the complexity level may be based on aduration or time requirement for completing the surgical procedure orportions thereof. For example, procedures or techniques requiring longerperformance times may be considered more complex and may be associatedwith a higher complexity level. As another example, the complexity levelmay be based on the number of steps required to perform the surgicalprocedure or portions thereof. For example, procedures or techniquesrequiring more steps may be considered more complex and may beassociated with a higher complexity level. In some embodiments, thecomplexity level may be based on the type of surgical techniques orprocedures being performed. Certain techniques or procedures may have apredetermined complexity and the complexity level may be based on thecomplexity of the techniques or procedures involved. For example, acholecystectomy may be considered more complex than an omentectomy and,accordingly, surgical procedures involving the cholecystectomy may beassigned a higher complexity level. Other factors that may be relevantto a complexity level may include information relating to diseaseseverity, complicating factors, anatomical structures involved, types ofmedical tools used, types of surgical procedures performed,intraoperative events (including adverse events) occurring in theprocedures, a physiological response of the patient, a medical conditionexhibited by the patient, patient characteristics, surgeoncharacteristics, a skill level of a surgeon or other healthcare providerinvolved, timing information (e.g., duration of interactions betweenmedical tools and anatomical structures, a duration of a surgical phaseor intraoperative event, time between appearance of a medical tool and afirst interaction between the medical tool and an anatomical structure,or other relevant duration or timing information), a condition of ananatomical structure, a number of surgeons or other healthcareprofessionals involved, or any other information associated with theprior surgical procedures. A surgical complexity level may not belimited to any of the examples above and may be based on a combinationof factors, including the examples provided above.

The surgical complexity level may be represented in various manners. Insome embodiments, the complexity level may be represented as a value.For example, the surgical complexity level may be a value within a rangeof values corresponding to a scale of complexity (e.g., 0-5, 0-10,0-100, or any other suitable scale). A percentage or other score mayalso be used. Generally, a higher value may indicate a higher complexitylevel, however, in some embodiments, the surgical complexity may be aninverse of the value. For example, a complexity level of 1 may indicatea higher complexity than a complexity level of 7. In other embodiments,the complexity level may be represented as a text-based indicator ofcomplexity. For example, the first set of frames may be assigned acomplexity level of “high complexity,” “moderate complexity,” “lowcomplexity,” or various other classifiers. In some embodiments, thesurgical complexity level may correspond to a standardized scale orindex used to represent surgical complexities. The surgical complexitylevel may be specific to a particular type of surgical procedure (or asubset of surgical procedure types), or may be a universal complexitylevel applicable to any surgical procedure.

As mentioned above, the first surgical complexity level may bedetermined by analyzing the first set of frames using historical data.Analyzing the first set of frames may include any process fordetermining the complexity level based on information included in thefirst set of frames. Examples of analysis for determining surgicalcomplexity levels are provided in greater detail below.

Further, the first surgical complexity level may be determined using theidentified anatomical structure. In some embodiments, a type ofanatomical structure involved in the procedure may be at least partiallyindicative of the surgical complexity level. For example, proceduresperformed on certain anatomical structures (e.g., anatomical structuresassociated with the brain or heart of a patient) may be considered morecomplex. In some embodiments, the condition of the anatomical structuremay also be relevant to determining the complexity level, as discussedin more detail below.

Some aspects of the present disclosure may involve analyzing frames ofthe surgical footage to identify in a second set of frames a medicaltool, the anatomical structure, and an interaction between the medicaltool and the anatomical structure. For example, the second set of framesmay indicate a portion of the surgical footage in which a surgicaloperation is being performed on the anatomical structure. A medical toolmay include any apparatus or equipment used as part of a medicalprocedure. In some embodiments, the medical tool may be a surgical tool,as discussed above. For example, the medical tool may include, but isnot limited to, cutting instruments, grasping and/or holdinginstruments, retractors, tissue unifying instruments and/or materials,protective equipment, laparoscopes, endoscopes, patient monitoringdevices, patient imaging devices, or similar tools. As discussed above,the interaction may include any action by the medical instrument thatmay influence the anatomical structure, or vice versa. For example, theinteraction may include a contact between the medical instrument and theanatomical structure, an action by the medical instrument on theanatomical structure (such as cutting, clamping, grasping, applyingpressure, scraping, etc.), a physiological response by the anatomicalstructure, or any other form of interaction.

As with the first set of frames, the second set of frames may be agrouping of one or more frames included within the surgical footage. Thesecond set of frames may be consecutive frames, or may include aplurality of groups of consecutive frames. In some embodiments, thefirst set of frames and the second set of frames may be completelydistinct. In other embodiments, the first set of frames and the secondset of frames may include at least one common frame appearing in boththe first set of frames and the second set of frames. As with the firstset of frames, the second set of frames may be analyzed to identify themedical tool, the anatomical structure, and the interaction between themedical tool and the anatomical structure using various techniques. Insome embodiments, the frames of the surgical footage may be analyzedusing object detection algorithms (e.g. appearance-based detectionalgorithms, image feature based detection algorithms, template baseddetection algorithms, etc.) and/or motion detection algorithms. In someembodiments, identifying the medical tool, the anatomical structure, andthe interaction between the medical tool and the anatomical structure inthe second set of frames may include using a machine learning modeltrained to detect medical tools, anatomical structures, and interactionsbetween medical tools and anatomical structures. For example, a machinelearning model may be trained using training examples to detect medicaltools and/or anatomical structures and/or interactions between medicaltools and anatomical structures from images and/or videos, and thetrained machine learning model may be used to analyze the second set offrames to detect the medical tools and/or the anatomical structuresand/or the interactions between medical tools and anatomical structures.An example of such training example may include an image and/or a videoclip of a surgical procedure, together with a label indicating at leastone of a medical tool depicted in the image and/or in the video clip, ananatomical structure depicted in the image and/or in the video clip, andan interaction between a medical tool and an anatomical structuredepicted in the image and/or in the video clip.

In some exemplary embodiments, identifying the anatomical structure inthe first set of frames may be based on an identification of a medicaltool and a first interaction between the medical tool and the anatomicalstructure. In some embodiments, the medical tool identified in the firstset of frames may be the same tool as the medical tool identified in thesecond set of frames. Accordingly, the interaction between the medicaltool and the anatomical structure in the second set of frames may be alater interaction between the medical tool and the anatomical structure.This may be helpful, for example, in determining a time between thefirst interaction and the later interaction, which may be at leastpartially indicative of a surgical complexity level.

Embodiments of the present disclosure may further include accessingsecond historical data, the second historical data being based on ananalysis of second frame data captured from a second group of priorsurgical procedures. In some embodiments, the first group of priorsurgical procedures and the second group of prior surgical proceduresmay be of a same type. For example, first historical data and secondhistorical data may relate to a first group of appendectomies and asecond group of appendectomies, respectively. A first group and secondgroup may differ according to a characteristic. By way of onenon-limiting example, the first group may involve patients exhibitingperitonitis, and the second group may include patients who did notexhibit peritonitis.

In some embodiments, first frame data and second frame data may beidentical (i.e., the first historical data and the second historicaldata may be based on the same frame data). For example, first historicaldata and second historical data may be based on different analysis ofthe same frame data. As an illustrative example, first frame data mayinclude estimates of surgical contact force not included in second framedata, consistent with the present embodiments. In some embodiments,first historical data and second historical data may be based ondifferent subsets of the same frame data (e.g., different surgicalphases and/or different surgical procedures).

In some embodiments, the first frame data and the second frame data maybe different (i.e., accessed or stored in different data structures).For example, different frames of the same surgical procedures may beanalyzed to generate the first historical data than the secondhistorical data.

In other embodiments the first group of prior surgical procedures andthe second group of prior surgical procedures may be different in atleast one aspect. For example, the first and second group may includeappendectomies but may differ in that the first group includesappendectomies in which an abnormal fluid leakage event was detectedwhile no abnormal fluid leakage events were detected in the secondgroup. In some embodiments, the first group of prior surgical proceduresand the second group of prior surgical procedures may have at least onesurgical procedure in common (e.g., both groups may include anincision). In other embodiments, however, the first group of priorsurgical procedures and the second group of prior surgical proceduresmay have no surgical procedures in common.

In some embodiments, a method may include tagging a first set of frameswith a first complexity level, tagging a second set of frames with thesecond complexity level, and storing first set of frames with the firsttag and the second set of frames with the second tag in a datastructure. This may enable a surgeon to select the second complexitylevel, and thereby cause the second set of frames to be displayed, whileomitting a display of the first set of frames. In some embodiments, amethod may include receiving a selection of a complexity level (e.g.,receiving a selection based on user input to an interface). Further, amethod may include accessing a data structure to retrieve selectedframes. A method may include displaying frames tagged with the selectedcomplexity level while omitting frames tagged without the selectedcomplexity level.

Similar to the first historical data and frame data, the secondhistorical data and frame data may be stored in one or more storagelocations. In some embodiments, the second historical data may be storedin the same storage location as the first historical data. In otherembodiments, the first and second historical data may be stored inseparate locations. Consistent with other embodiments, accessing thefirst historical data may include receiving the second historical dataand/or the second frame data, for example from an image capture deviceor a computing device. Further as with the first historical data,accessing the second historical data may include retrieving or receivingthe second frame data and analyzing the second frame data to identifythe second historical data. In some embodiments, the first historicaldata and the second historical data may be identical. In otherembodiments, the first historical data and the second historical datamay be different. The second historical data may include informationpertaining to the second frame data, similar to the first historicaldata, as discussed above. The second historical data may include any ofthe information described above with respect to the first historicaldata, such as medical tool information, anatomical structureinformation, and/or associated complexity information. In embodimentswhere the second historical data includes complexity information, suchinformation may be indicative of or associated with the complexityinformation. For example, the second historical data may include anindication of a statistical relation between a particular anatomicalstructure and a particular surgical complexity level.

Some aspects of the present disclosure may involve analyzing the secondset of frames using the second historical data and using the identifiedinteraction to determine a second surgical complexity level associatedwith the second set of frames. The second surgical complexity level maybe similar to the first surgical complexity level and thus may be basedon one or more of the example factors provided above with respect to thefirst surgical complexity level. In some embodiments, the secondsurgical complexity level may be represented in the same form as thefirst surgical complexity level (e.g., as a value within the same scale,etc.), however, a different form of representation may be used in someembodiments.

Consistent with embodiments of the present disclosure, the first andsecond surgical complexity levels may be determined according to variousmethods. In some embodiments, the disclosed embodiments may includeusing a machine learning model trained to identify surgical complexitylevels using frame data captured from prior surgical procedures todetermine at least one of the first surgical complexity level or thesecond surgical complexity level. For example, a machine learning modelmay be developed using a machine learning algorithm. Training data,which may include frame data captured from prior surgical procedures andlabels indicating surgical complexity levels known to correspond to theframe data, may be supplied to a machine learning algorithm to developthe trained model. The machine learning algorithm may include a logisticregression, a linear regression, a regression, a random forest, aK-Nearest Neighbor (KNN) model, a K-Means model, a decision tree, a coxproportional hazards regression model, a Naïve Bayes model, a SupportVector Machines (SVM) model, an artificial neural network, a gradientboosting algorithm, or any other form of machine learning model oralgorithm. Accordingly, the first historical data may include a machinelearning model trained using the first frame data captured from thefirst group of prior surgical procedures. Similarly, the secondhistorical data may comprise a machine learning model trained using thesecond frame data captured from the second group of prior surgicalprocedures. As a result, the trained model, when provided the first setof frames and the second set of frames, may be configured to determinethe first and second surgical complexity levels, respectively.

In some exemplary embodiments, at least one of determining the firstcomplexity level or second complexity level may be based on aphysiological response. As discussed above, the physiological responsemay include any physical or anatomical condition or reaction of thepatient resulting, either directly or indirectly, from the surgicalprocedure. For example, the physiological response may include, a changein heart rate, a physical movement, a failure or decrease in function ofone or more organs, a change in body temperature, a spoken reaction ofthe patient, a change in brain activity, a change in respiratory rate, achange in perspiration, a change in blood oxygen level, a change inheart function, activation of the sympathetic nervous system, anendocrine response, cytokine production, acute phase reaction,neutrophil leukocytosis, lymphocyte proliferation, or any other physicalchange in response to the surgical procedure. In some embodiments, thephysiological response may be indicative of the surgical complexitylevel. For example, surgical procedures that trigger a certainphysiological response may be considered more complex and thus may havea higher complexity level rating. For example, a machine learning modelmay be trained using training examples to identify physiologicalresponses from images and/or videos, the trained machine learning modelmay be used to analyze the first set of frames to identify a firstphysiological response and/or to analyze the second set of frames toidentify a second physiological response, and the first surgicalcomplexity level may be determined based on the identified firstphysiological response and/or the second surgical complexity level maybe determined based on the identified second physiological response. Anexample of such training example may include an image and/or a videoclip of a surgical procedure, together with a label indicating aphysiological response depicted in the image and/or the video clip.

In some exemplary embodiments, determining at least one of the firstsurgical complexity level or the second surgical complexity level may bebased on a condition of the anatomical structure, as mentioned above. Byway of example, the condition may involve a detected deterioration ofthe anatomical structure, a tear, bleeding, swelling, discoloration,distortion, or any properties of the anatomical structure reflective ofits current state. In some embodiments, the condition of the anatomicalstructure may include a medical condition affecting the anatomicalstructure. This medical condition may indicate the purpose or type ofsurgical procedure being performed and thus may indicate an associatedcomplexity level. For example, if a gallbladder exhibits large polyps,this may indicate that a cholecystectomy is involved in the surgicalprocedure, which may be useful for determining the complexity level. Inother embodiments, the medical condition may indicate one or morecomplicating factors associated with the surgical procedure. Forexample, hemorrhaging occurring at the anatomical structure may indicatecomplications that have arisen during the surgical procedure, which mayaffect the surgical complexity level. Alternatively, or additionally,the medical condition itself may be associated with a certain complexitylevel. In some embodiments, the condition of the anatomical structuremay be a state of the anatomical structure based on the current stage orphase of the surgical procedure. For example, an incision made in theanatomical structure may impact the condition of the anatomicalstructure and thus change a complexity level as compared to a complexitylevel before the incision. For example, a machine learning model may betrained using training examples to identify condition of anatomicalstructures from images and/or videos, the trained machine learning modelmay be used to analyze the first set of frames to identify a firstcondition of a first anatomical structure and/or to analyze the secondset of frames to identify a second condition of a second anatomicalstructure (while may be the same as the first anatomical structure or adifferent anatomical structure), and the first surgical complexity levelmay be determined based on the identified first condition and/or thesecond surgical complexity level may be determined based on theidentified second condition. An example of such training example mayinclude an image and/or a video clip of an anatomical structure,together with a label indicating a condition of the anatomicalstructure.

In some embodiments of the present disclosure, determining at least oneof the first surgical complexity level or the second surgical complexitylevel may be based on a patient characteristic. Patient characteristicsmay include, but are not limited to, age, gender, weight, height, BodyMass Index (BMI), menopausal status, typical blood pressure,characteristics of the patient genome, educational status, level ofeducation, economical status, level of income, level of occupation, typeof insurance, health status, self-rated health, functional status,functional impairment, duration of disease, severity of disease, numberof illnesses, illness characteristics (such as type of illness, size oftumor, histology grade, number of infiltrated lymph nodes, etc.),utilization of health care, number of medical care visits, medical carevisit intervals, regular source of medical care, family situation,marital status, number of children, family support, ethnicity, race,acculturation, religious, type of religion, native language,characteristics of past medical test performed on the patient (such astype of test, time of test, results of test, etc.), characteristics ofpast medical treatments performed on the patient (such as type oftreatment, time of treatment, results of treatment, etc.), or any otherrelevant characteristic. Other example patient characteristics aredescribed throughout the present disclosure. These characteristics maybe correlated with certain levels of surgical complexity. For example,an older and/or overweight patient may be associated with surgicalprocedures having higher complexities than patients that are younger orin better physical shape.

In accordance with some embodiments, determining at least one of thefirst surgical complexity level or the second surgical complexity levelmay be based on a skill level of a surgeon associated with the surgicalfootage. For example, if a surgeon depicted in surgical footage has alow skill level, then a procedure that might ordinarily be considered ashaving a low complexity may be made more complex as the result of thereduced performance skill. Thus, as discussed above, the skill level maybe an indication of the surgeon's ability to perform the surgicalprocedure or specific techniques within the surgical procedure. In someembodiments, the skill level may relate to past performances of thesurgeon, a type and/or level of training or education the surgeon hasreceived, a number of surgeries the surgeon has performed, types ofsurgeries surgeon has performed, qualifications of the surgeon, years ofexperience of the surgeon, ratings of the surgeon from patients or otherhealthcare professionals, past surgical outcomes, past surgicalcomplications, or any other information relevant to assessing the skilllevel of a surgeon. Alternatively or additionally, the skill level ofthe surgeon may be determined through computer analysis of videofootage. For example, artificial intelligence can be used to classify asurgeon's skill level, as discussed in greater detail below. While theskill level is described herein as the skill level of a surgeon, in someembodiments the skill level may be associated with another healthcareprofessional, such as anesthesiologists, nurses, Certified RegisteredNurse Anesthetist (CRNA), surgical technicians, residents, medicalstudents, physician assistants, or any other healthcare professional.Thus, reference to a surgeon as used throughout this disclosure is ashorthand for any relevant medical professional.

Some embodiments of the present disclosure may further includedetermining a level of skill demonstrated by a healthcare provider inthe surgical footage. At least one of determining the first complexitylevel or second complexity level may be based on the determined level ofskill demonstrated by the healthcare provider. The skill level of thehealthcare provider may be determined based on analysis of the first orsecond set of frames using image and/or video analysis algorithms, suchas object and/or motion detection algorithms. For example, thehealthcare provider may perform one or more techniques in a manner thatdemonstrates a certain level of skill. In one example, a machinelearning model may be trained using training examples to determine skilllevels of healthcare providers from images and/or videos, and thetrained machine learning model may be used to analyze the surgicalfootage and determine the level of skill demonstrated by the healthcareprovided in the surgical footage. An example of such training examplemay include a video clip depicting a portion of a surgical procedure,together with a label indicating the level of skill demonstrated in thevideo clip. In other embodiments, the skill level may be determinedbased on an identity of the healthcare provider in the surgical footage.For example, based on the identity of a surgeon, an associated skilllevel may be determined from an external source, such as a databaseincluding skill level information for various surgeons. Accordingly, oneor more facial recognition algorithms may be used to identify thehealthcare provider, and the identity of the healthcare provider may beused to determine the healthcare provider skill level.

In some exemplary embodiments, determining at least one of the firstsurgical complexity level or the second surgical complexity level may bebased on an analysis of an electronic medical record. In someembodiments, information regarding a medical history of the patient,which may be included in the electronic medical record, may be relevantto the complexity level of a surgical procedure being performed on thepatient. For example, the electronic medical record may include surgicalhistory (such a list of surgeries performed on the patient, operativereports, etc.), obstetric history (such as a list of pregnancies,possibly together with details associated with the pregnancies, such ascomplications, outcomes, etc.), allergies, past and present medications,immunization history, growth chart and/or development history, notesfrom past medical encounters (for example, such note may include detailsabout the complaints, physical examinations, medical assessment,diagnosis, etc.), test results, medical images (such as X-ray images,Computed Tomography images, Magnetic Resonance Imaging images, PositronEmission Tomography images, Single-Photon Emission Computed Tomographyimages, UltraSound images, Electro-Cardio-Graphy images,Electro-Encephalo-Graphy images, Electro-Myo-Graphy images,Magneto-Encephalo-Graphy images, etc.) and/or information based onmedical images, medical videos and/or information based on medicalvideos, orders, prescriptions, medical history of the patient's family,and so forth.

In accordance with embodiments of the present disclosure, determiningthe first surgical complexity level may further include identifying inthe first set of frames a medical tool. In some embodiments, the medicaltool identified in the first set of frames may correspond to the medicaltool identified in the second set of frames. For example, the same toolmay be identified in both sets of frames. In other embodiments, themedical tool identified in the first set of frames may differ from themedical tool identified in the second set of frames. Determining thefirst surgical complexity level may be based on a type of the medicaltool. The type of tool appearing in the first set of frames may beindicative of the type and/or complexity of procedure being performed.For example, if the medical tool is a specialized tool, used only forcertain procedures or types of procedures, the complexity level may bedetermined at least in part based on the complexity associated withthose procedures or types of procedures.

In some exemplary embodiments, determining the first surgical complexitylevel may be based on an event that occurred after the first set offrames. For example a surgical event such as a leak that occurs inframes after a first set of frames depicting suturing, may inform thecomplexity level associated with the first set of frames. (e.g., thesuturing procedure that might otherwise be associated with a lowercomplexity level based on the first set of frames alone, may be elevatedto a higher complexity level when from the footage it was determinedthat the leak likely occurred as the result of improper suturing. Thelater event may include any event related to the surgical procedure thathas an impact on a surgical complexity of the footage, including thevarious examples of intraoperative surgical events described throughoutthe present disclosure. By way of another example, the event thatoccurred after the first set of frames may be an adverse event, such asa bleed, that occurs after the first set of frames. The occurrence ofthe event may provide context for determining the first surgicalcomplexity level. In some embodiments, the event occurring after thefirst set of frames may be identified based on analysis of additionalframes. For example, the event may occur before the second set of framesand may be identified based on analyzing frames between the first set offrames and the second set of frames. In other embodiments, theoccurrence of the event between the first and second set of frames maybe inferred based on the second set of frames, without analyzingadditional frames. Further, in some embodiments the event may occurafter the second set of frames.

Similarly, in some embodiments, determining the second surgicalcomplexity level may be based on an event that occurred between thefirst set of frames and the second set of frames. The event may occur atother times, including at the first set of frames, before the first setof frames, or after the second set of frames. In some embodiments, thefirst and/or second surgical complexity level may be determined based onoccurrence of the event based on a machine learning model trained tocorrelate events and/or event timings with various complexity levels. Asan illustrative example, determining the second surgical complexitylevel may be based on an indication that an additional surgeon wascalled after the first set of frames. The indication that an additionalsurgeon was called may include, for example, the presence of a surgeonin the second set of frames but in first set of frames. Calling of theadditional surgeon may indicate that the surgeon performing the surgeryneeded assistance and/or guidance, which may be relevant to determiningthe surgical complexity level. In another example, determining thesecond surgical complexity level may be based on an indication that aparticular medicine was administered after the first set of frames. Forexample, the medicine may include an anesthesia (e.g., local, regional,and/or general anesthesia), a barbiturate, a benzodiazepine, a sedative,a coagulant, or various other medications that may be administeredduring a surgical procedure. Administration of the medicine may berelevant to determining the surgical complexity level. In someembodiments, administration of the medicine may be indicative of one ormore complications that may have occurred, which may also be relevantdetermining the surgical complexity level.

In accordance with the embodiments of the present disclosure determiningthe second surgical complexity level may be based on time elapsed fromthe first set of frames to the second set of frames. For example, thetime elapsed from the first set of frames to the second set of framesmay represent a time between when an anatomical structure first appearsin the surgical footage and the first time a medical tool interacts withthe anatomical structure. As another example, the elapsed time mayindicate the time between two surgical phases and/or intraoperativesurgical events. For example, in embodiments where determining the firstsurgical complexity level further includes identifying in the first setof frames a medical tool, the first set of frames may indicate onesurgical phase, such as an incision, and the second set of frames mayindicate a second surgical phase, such as a suturing. The elapsed timebetween the two phases or events may be at least partially indicative ofa surgical complexity level. (E.g., an elapsed time greater than normalfor a particular procedure may indicate that the procedure was morecomplex than normal.) Other time durations within the surgical proceduremay also be indicative of the surgical complexity level, such as aduration of an action, a duration of an event, a duration of a surgicalphase, a duration between an action and a corresponding physiologicalresponse, and so forth. The surgical footage may be analyzed to measuresuch time durations, and the determination of the surgical complexitylevels may be based on the determined time durations.

Embodiments of the present disclosure may further include comparing thefirst and/or second surgical complexity levels to a selected threshold.In some embodiments, the selected threshold may be used to select whichframes should be selected for display and/or inclusion in a datastructure. For example, the disclosed methods may include determiningthat the first surgical complexity level is less than a selectedthreshold and determining that the second surgical complexity levelexceeds the selected threshold. This may indicate that the second set offrames are associated with a complexity level meeting a minimumcomplexity level, while the first set of frames are not. Accordingly,the disclosed methods may further include, in response to thedetermination that the first surgical complexity level is less than theselected threshold and the determination that the second surgicalcomplexity level exceeds the selected threshold, storing the second setof frames in a data structure while omitting the first set of framesfrom the data structure. The data structure may be used by a surgeon orother user for selecting video for display meeting a minimum complexitylevel requirement.

Some embodiments of the present disclosure may further include taggingthe first set of frames with the first surgical complexity level;tagging the second set of frames with the second surgical complexitylevel; and generating a data structure including the first set of frameswith the first tag and the second set of frames with the second tag. Thedata structure may associate the first and second set of frames, as wellas other frames of the surgical video footage, with the correspondingcomplexity level such that it is indexed for easy retrieval. Suchindexing may correspond to the video indexing discussed in detail above.For example, the surgical complexity level may be an eventcharacteristic as described above and as illustrated in data structure600, shown in FIG. 6 . Accordingly, generating the data structure mayenable a surgeon to select the second surgical complexity level, andthereby cause the second set of frames to be displayed, while omitting adisplay of the first set of frames. For example, video may be selectedfor playback based on process 800 described above with respect to FIGS.8A and 8B.

FIG. 13 is a flowchart illustrating an example process 1300 foranalyzing complexity of surgical footage, consistent with the disclosedembodiments. Process 1300 may be performed by at least one processingdevice, such as processor, as described herein. By way of one example aprocessor may include processors 1412 as illustrated in FIG. 14 .Throughout this disclosure, the term “processor” is used as a shorthandfor “at least one processor.” In other words, a processor may includeone or more structures that perform logic operations whether suchstructures are collocated, connected, or disbursed. In some embodiments,a non-transitory computer readable medium may contain instructions thatwhen executed by a processor cause the processor to perform process1300. Process 1300 is not necessarily limited to the steps shown in FIG.1300 , and any steps or processes of the various embodiments describedthroughout the present disclosure may also be included in process 1300.At step 1310, process 1300 may include analyzing frames of the surgicalfootage to identify in a first set of frames an anatomical structure, asdiscussed previously. In some embodiments, the anatomical structure maybe identified using an image and/or video analysis algorithm, such as anobject or motion detection algorithm, as previously discussed. In otherembodiments, the anatomical structure may be identified using a machinelearning model trained to detect anatomical structures, as describedearlier.

At step 1320, process 1300 may include accessing first historical data,the first historical data being based on an analysis of first frame datacaptured from a first group of prior surgical procedures. In someembodiments, the first historical data may include a machine learningmodel trained using the first frame data captured from the first groupof prior surgical procedures, as described previously. At step 1330,process 1300 may include analyzing the first set of frames using thefirst historical data and using the identified anatomical structure todetermine a first surgical complexity level associated with the firstset of frames. For example, a machine learning model may be trainedusing training data (for example, training data based on the historicaldata based on an analysis of frame data captured from prior surgicalprocedures) to identify surgical complexity level associated with a setof frames, and the trained machine learning model may be used to analyzethe first set of frames to determine a first surgical complexity levelassociated with the first set of frames.

At step 1340, process 1300 may include analyzing frames of the surgicalfootage to identify in a second set of frames a medical tool, theanatomical structure, and an interaction between the medical tool andthe anatomical structure, as described in greater detail previously. Forexample, object detection algorithms and/or action detection algorithmsmay be used to analyze the second set of frames to detect the medicaltool and/or the anatomical structure and/or the interaction between themedical tool and the anatomical structure. In another example, a machinelearning model trained using training examples to detect medical toolsand/or anatomical structures and/or the interaction between the medicaltools and the anatomical structures in images and/or videos may be used.At step 1350, process 1300 may include accessing second historical data,the second historical data being based on an analysis of second framedata captured from a second group of prior surgical procedures. In someembodiments, the first historical data and the second historical datamay be identical. In other embodiments, the first historical data andthe second historical data may be different. At step 1360, process 1300may include analyzing the second set of frames using the secondhistorical data and using the identified interaction to determine asecond surgical complexity level associated with the second set offrames, as previously described.

An operating room schedule may need to be adjusted based on delaysassociated with surgical Disclosed systems and methods may involveanalyzing surgical footage to identify features of surgery, patientconditions, and other features to determine adjustments to an operatingroom schedule, procedures conducted in the operating room. Conversely,the schedule may need to be adjusted if a surgical procedure iscompleted ahead of a scheduled time. Therefore, there is a need foradjusting an operating room schedule in an effective and efficientmanner using information obtained from surgical footage during asurgical procedure

Aspects of this disclosure may relate to adjusting an operating roomschedule, including methods, systems, devices, and computer-readablemedia. The operating room schedule may include a scheduled timeassociated with completion of the ongoing surgical procedure, as well asscheduled times for starting and finishing future surgical procedures.

Both a method for enabling adjustments of an operating room schedule anda system is described below, with the understanding that aspects of themethod or the system may occur electronically over a network that iseither wired, wireless, or both. Other aspects of such a method orsystem may occur using non-electronic means. In the broadest sense, themethod or the system is not limited to a particular physical and/orelectronic instrumentality, but rather may be accomplished using manydiffering instrumentalities. For ease of discussion, a method isdescribed first below, with the understanding that aspects of the methodapply equally to systems, devices, and computer-readable media.

Disclosed embodiments may involve receiving from an image sensorpositioned in a surgical operating room, visual data tracking an ongoingsurgical procedure. As used herein, the visual data may include any formof recorded visual media, including recorded images, one or more framesor images or clips, and/or data directly or indirectly derived from theforegoing. Additionally, the video data may include sound. For example,the visual data may include a sequence of one or more images captured byimage sensors, such as cameras 115, 121, 123, and/or 125, as describedabove in connection with FIG. 1 . Some of the cameras (e.g., cameras115, 121 and 125) may capture video/image data of operating table 141,camera 121 may capture video/image data of a surgeon 131 performing thesurgery. In some cases, cameras may capture video/image data associatedwith surgical team personnel, such as anesthesiologists, nurses,surgical technicians, or other healthcare professionals located inoperating room 101.

In various embodiments, image sensors may be configured to capturevisual data by converting visible light, x-ray light (e.g., viafluoroscopy), infrared light, or ultraviolet light to images, sequenceof images, videos, and any other form of representations. Theimage/video data may be stored as computer files using any suitableformat such as JPEG, PNG. TIFF, Audio Video Interleave (AVI), FlashVideo Format (FLV), QuickTime File Format (MOV), MPEG (MPG, MP4. M4P,etc.), Windows Media Video (WMV), Material Exchange Format (MXF),uncompressed formats, lossless compressed formats, lossy compressedformats, or other audio or video format.

An image sensor may be any sensor capable of capturing image or videodata. A single sensor may be used, or multiple image sensors may bepositioned in a surgical operating room (e.g., the sensors may bepositioned throughout the operating room). In an illustrativeembodiment, an example image sensor may be positioned above a patient.The example image sensor may be above an operating table, next to theoperating table, next to devices located in the operating room, oranywhere else capable of detecting information about a surgery. As shownin FIG. 1 , the image sensor may include cameras 115-125. In some cases,image sensors may be wearable devices (e.g., head mounted cameras, bodymounted cameras, or any sensor capable of being associated with aperson). Additionally or alternatively, an example image sensor may bepositioned on a surgical tool (i.e., be a part of a surgicalinstrument). For example, an image sensor may be a part of abronchoscope tube, a laparoscope, an endoscope, or any other medicalinstrument configured for location inside or outside a patient (e.g.,for procedures such as gastroscopy, colonoscopy, hysteroscopy,cystoscopy, flexible sigmoidoscopy, wireless capsule endoscopy, and thelike).

Image sensors, particularly when being part of surgical instruments, mayinclude one or more light emitting sources for emitting light ofsuitable wavelength such as visible light, infrared light, and/orultraviolet light. The light emitting sources may include any suitablesources (e.g., light emitting diodes (LEDs) emitting visible light,fluorescent light sources, incandescent light sources, infrared LEDs,ultraviolet LEDs, and/or other type of light source). Image sensors maynot be limited to capturing light, but may be configured to processother signals for producing visual data related to the captured signals.For example, image sensors may be configured to capture ultrasound,changes in an electromagnetic field, or any other suitable signals(e.g., distribution of a force over a surface), and the like to producevisual data related to the captured signals.

A surgical procedure may include any medical procedure associated withor involving manual or operative procedures on a patient's body.Surgical procedures may include cutting, abrading, suturing, and/orother techniques that involve measuring, treating or physically changingbody tissues and/or organs. Some non-limiting examples of such surgicalprocedures may include a laparoscopic surgery, a thoracoscopicprocedure, a bronchoscopic procedure, a microscopic procedure, an opensurgery, a robotic surgery, an appendectomy, a carotid endarterectomy, acarpal tunnel release, a cataract surgery, a cesarean section, acholecystectomy, a colectomy (such as a partial colectomy, a totalcolectomy, etc.), a coronary angioplasty, a coronary artery bypass, adebridement (for example of a wound, a burn, an infection, etc.), a freeskin graft, a hemorrhoidectomy, a hip replacement, a hysterectomy, ahysteroscopy, an inguinal hernia repair, a knee arthroscopy, a kneereplacement, a mastectomy (such as a partial mastectomy, a totalmastectomy, a modified radical mastectomy, etc.), a prostate resection,a prostate removal, a shoulder arthroscopy, a spine surgery (such as aspinal fusion, a laminectomy, a foraminotomy, a diskectomy, a diskreplacement, an interlaminar implant, etc.), a tonsillectomy, a cochlearimplant procedure, brain tumor (for example meningioma, etc.) resection,interventional procedures such as percutaneous transluminal coronaryangioplasty, transcatheter aortic valve replacement, minimally invasivesurgery for intracerebral hemorrhage evacuation, or any other medicalprocedure involving some form of incision. While the present disclosureis described in reference to surgical procedures, it is to be understoodthat it may also apply to other forms of medical procedures orprocedures generally.

An operating room may be any suitable facility (e.g., a room within ahospital) where surgical operations are carried out in an asepticenvironment. The operating room may be configured to be well-lit and tohave overhead surgical lights. The operating room may feature controlledtemperature and humidity and may be windowless. In an exampleembodiment, the operating room may include air handlers that may filterthe air and maintain a slightly elevated pressure within the operatingroom to prevent contamination. The operating room may include anelectricity backup system in case of a black-out and may include asupply of oxygen and anesthetic gases. The room may include a storagespace for common surgical supplies, containers for disposables, ananesthesia cart, an operating table, cameras, monitors, and/or otheritems for surgery. A dedicated scrubbing area that is used by surgeons,anesthetists, operating department practitioners (ODPs), and nursesprior to surgery may be part of the operating room. Additionally, a mapincluded in the operating room may enable the terminal cleaner torealign the operating table and equipment to the desired layout duringcleaning. In various embodiments, one or more operating rooms may be apart of an operating suite that may form a distinct section within ahealthcare facility. The operating suite may include one or morewashrooms, preparation and recovery rooms, storage and cleaningfacilities, offices, dedicated corridors, and possibly other supportiveunits. In various embodiments, the operating suite may be climate- andair-controlled, and separated from other departments.

In various embodiments, visual data captured by image sensors may trackan ongoing surgical procedure. In some cases, visual data may be used totrack a region of interest (ROI) such as a region of a body of a patientin which an operation is conducted (e.g., a region 127, as shown in FIG.1 ). In an example embodiment, cameras 115-125 may capture visual databy tracking the ROI via camera motion, camera rotation, or by zoomingtowards the ROI. For instance, camera 115 may be movable and point atthe ROI, at which video/image data needs to be captured during, before,or after a surgical procedure. For example, as shown in FIG. 1 , camera115 may be rotated as indicated by arrows 135A showing a pitchdirection, and arrows 135B showing a yaw direction for camera 115. Invarious embodiments, pitch and yaw angles of cameras (e.g., camera 115)may be controlled to track the ROI.

In an example embodiment, camera 115 may be configured to track asurgical instrument (also referred to as a surgical tool, a medicalinstrument, etc.) within location 127, an anatomical structure, a handof surgeon 131, an incision, a movement of anatomical structure, and/orany other object. In various embodiments, camera 115 may be equippedwith a laser 137 (e.g., an infrared laser) for precision tracking. Insome cases, camera 115 may be tracked automatically via a computer basedcontrol application that uses an image recognition algorithm forpositioning the camera to capture video/image data of the ROI. Forexample, the control application may identify an anatomical structure,identify a surgical tool, hand of a surgeon, bleeding, motion, and thelike at a particular location within the anatomical structure, and trackthat location with camera 115 by rotating camera 115 by appropriate yawand pitch angles. In some embodiments, the control application maycontrol positions (i.e., yaw and pitch angles) of various cameras115-125 to capture video/image date from more than one ROI during asurgical procedure. Additionally or alternatively, a human operator maycontrol the position of various cameras 115-125, and/or the humanoperator may supervise the control application in controlling theposition of the cameras.

As used herein, the term “anatomical structure” may include anyparticular part of a living organism, including, for example, one ormore organs, tissues, ducts, arteries, cells, or any other anatomicalparts. In some cases, prosthetics, artificial organs, and the like maybe considered as anatomical structures.

Cameras 115-125 may further include zoom lenses for magnifying one ormore ROIs. In an example embodiment, camera 115 may include a zoom lens138 for magnifying a ROI (e.g., a surgical tool in the proximity of ananatomical structure). Camera 121 may include a zoom lens 139 forcapturing video/image data from a larger area around the ROI. Forexample, camera 121 may capture video/image data for the entire location127. In some embodiments, video/image data obtained from camera 121 maybe analyzed to identify a ROI during the surgical procedure, and thecontrol application may be configured to cause camera 115 to zoomtowards the ROI identified by camera 121.

In various embodiments, the control application may be configured tocoordinate the position and zoom of various cameras during a surgicalprocedure. For example, the control application may direct camera 115 tovisually track an anatomical structure, and may direct camera 121 and125 to track a surgical instrument. Cameras 121 and 125 may track thesame ROI (e.g., a surgical instrument) from different view angles. Forexample, video/image data obtained from different view angles may beused to determine the position of the surgical instrument relative to asurface of the anatomical structure.

In various embodiments, control of position and zoom of cameras 115-125may be rule-based and follow an algorithm developed for a given surgicalprocedure. For example, the control application may be configured todirect camera 115 to track a surgical instrument, to direct camera 121to location 127, to direct camera 123 to track the motion of thesurgeon's hands, and to direct camera 125 to an anatomical structure.The algorithm may include any suitable logical statements determiningposition and zoom (magnification) for cameras 115-125 depending onvarious events during the surgical procedure. For example, the algorithmmay direct at least one camera to a region of an anatomical structurethat develops bleeding during the procedure.

In various cases, when a camera (e.g., camera 115) tracks a moving ordeforming object (e.g., when camera 115 tracks a moving surgicalinstrument, or a moving/pulsating anatomical structure) the controlapplication may determine a maximum allowable zoom for camera 115, suchthat the moving or deforming object does not escape a field of view ofthe camera. In an example embodiment, the control application mayinitially select the first zoom for camera 115, evaluate whether themoving or deforming object escapes the field of view of the camera, andadjust the zoom of the camera as necessary to prevent the moving ordeforming object from escaping the field of view of the camera. Invarious embodiments, the camera zoom may be readjusted based on adirection and a speed of the moving or deforming object. In some cases,the control application may be configured to predict future position andorientation of cameras 115-125 based on the movement of the hand of thesurgeon, the movement of a surgical instrument, the movement of a bodyof the surgeon, historical data reflecting likely next steps, or anyother data from which future movement may be derived.

The visual data captured by image sensors may be communicated via anetwork to a computer system for further analysis and storage. Forexample. FIG. 14 shows an example system 1401 that may include acomputer system 1410, a network 1418, and image sensors 1421 (e.g.,cameras positioned within the operating room), and 1423 (e.g., imagesensors being part of a surgical instrument) connected via network 1418to computer system 1401. System 1401 may include a database 1411 forstoring various types of data related to previously conducted surgeries(i.e., historical surgical data that may include historical image, videoor audio data, text data, doctors' notes, data obtained by analyzinghistorical surgical data, and other data relating to historicalsurgeries). In various embodiments, historical surgical data may be anysurgical data related to previously conducted surgical procedures.Additionally, system 1401 may include one or more audio sensors 1425,light emitting devices 1427, and a schedule 1430.

Computer system 1410 may include one or more processors 1412 foranalyzing the visual data collected by the image sensors, a data storage1413 for storing the visual data and/or other types of information, aninput module 1414 for entering any suitable input for computer system1410, and software instructions 1416 for controlling various aspects ofoperations of computer system 1410.

One or more processors 1412 of system 1410 may include multiple coreprocessors to handle concurrently multiple operations and/or streams.For example, processors 1412 may be parallel processing units toconcurrently handle visual data from different image sensors 1421 and1423. In some embodiments, processors 1412 may include one or moreprocessing devices, such as, but not limited to, microprocessors fromthe Pentium™ or Xeon™ family manufactured by Intel™, the Turion™ familymanufactured by AMD™, or any of various processors from othermanufacturers. Processors 1412 may include a plurality of co-processors,each configured to run specific operations such as floating-pointarithmetic, graphics, signal processing, string processing, or 1/Ointerfacing. In some embodiments, processors may include afield-programmable gate array (FPGA), central processing units (CPUs),graphical processing units (GPUs), and the like.

Database 1411 may include one or more computing devices configured withappropriate software to perform operations for providing content tosystem 1410. Database 1411 may include, for example, Oracle™ database,Sybase™ database, and/or other relational databases or non-relationaldatabases, such as Hadoop™ sequence files, HBase™, or Cassandra™. In anillustrative embodiment, database 1411 may include computing components(e.g., database management system, database server, etc.) configured toreceive and process requests for data stored in memory devices of thedatabase and to provide data from the database. As discussed before,database 1411 may be configured to collect and/or maintain the dataassociated with surgical procedures. Database 1411 may collect the datafrom a variety of sources, including, for instance, online resources.

Network 1418 may include any type of connections between variouscomputing components. For example, network 1418 may facilitate theexchange of information via network connections that may includeInternet connections, Local Area Network connections, near fieldcommunication (NFC), and/or other suitable connection(s) that enablesthe sending and receiving of information between the components ofsystem 1401. In some embodiments, one or more components of system 1401may communicate directly through one or more dedicated communicationlinks.

Various example embodiments of the system 1401 may includecomputer-implemented methods, tangible non-transitory computer-readablemediums, and systems. The computer-implemented methods may be executed,for example, by at least one processor that receives instructions from anon-transitory computer-readable storage medium such as medium 1413, asshown in FIG. 14 . Similarly, systems and devices consistent with thepresent disclosure may include at least one processor and memory, andthe memory may be a non-transitory computer-readable storage medium. Asused herein, a non-transitory computer-readable storage medium refers toany type of physical memory on which information or data readable by atleast one processor can be stored. Examples may include random accessmemory (RAM), read-only memory (ROM), volatile memory, non-volatilememory, hard drives, CD ROMs, DVDs, flash drives, disks, and any otherknown physical storage medium whether some or all portions thereof arephysically located in or near the operating room, in another room of thesame facility, at a remote captive site, or in a cloud-based serverfarm. Singular terms, such as “memory” and “computer-readable storagemedium,” may additionally refer to multiple structures, such a pluralityof memories or computer-readable storage mediums. As referred to herein,a “memory” may include any type of computer-readable storage mediumunless otherwise specified. A computer-readable storage medium may storeinstructions for execution by at least one processor, includinginstructions for causing the processor to perform steps or stagesconsistent with an embodiment herein. Additionally, one or morecomputer-readable storage mediums may be utilized in implementing acomputer-implemented method. The term “computer-readable storage medium”should be understood to include tangible items and exclude carrier wavesand transient signals.

Input module 1414 may be any suitable input interface for providinginput to one or more processors 1412. In an example embodiment, inputinterface may be a keyboard for inputting alphanumerical characters, amouse, a joystick, a touch screen, an on-screen keyboard, a smartphone,an audio capturing device (e.g., a microphone), a gesture capturingdevice (e.g., camera), and other device for inputting data. While a userinputs the information, the information may be displayed on a monitor toensure the correctness of the input. In various embodiments, the inputmay be analyzed verified or changed before being submitted to system1410.

Software instructions 1416 may be configured to control various aspectsof operation of system 1410, which may include receiving and analyzingthe visual data from the image sensors, controlling various aspects ofthe image sensors (e.g., moving image sensors, rotating image sensors,operating zoom lens of image sensors for zooming towards an example ROI,and/or other movements), controlling various aspects of other devices inthe operating room (e.g., controlling operation of audio sensors,chemical sensors, light emitting devices, and/or other devices).

As previously described, image sensors 1421 may be any suitable sensorscapable of capturing image or video data. For example, such sensors maybe cameras 115-125.

Audio sensors 1425 may be any suitable sensors for capturing audio data.Audio sensors 1425 may be configured to capture audio by convertingsounds to digital information. Some examples of audio sensors 1425 mayinclude microphones, unidirectional microphones, bidirectionalmicrophones, cardioid microphones, omnidirectional microphones, onboardmicrophones, wired microphones, wireless microphones, any combination ofthe above, and any other sound-capturing device.

Light emitting devices 1427 may be configured to emit light, forexample, in order to enable better image capturing by image sensors1421. In some embodiments, the emission of light may be coordinated withthe capturing operation of image sensors 1421. Additionally oralternatively, the emission of light may be continuous. In some cases,the emission of light may be performed at selected times. The emittedlight may be visible light, infrared light, ultraviolet light, deepultraviolet light, x-rays, gamma rays, and/or in any other portion ofthe light spectrum.

As described below, schedule 1430 may include an interface fordisplaying a scheduled time associated with completion of the ongoingsurgical procedure, as well as scheduled times for starting andfinishing future surgical procedures. Schedule 1430 may be implementedusing any suitable approach (e.g., as a standalone software application,as a website, as a spreadsheet, or any other suitable computer-basedapplication or a paper-based document). An example schedule 1430 mayinclude a list of procedures and list of starting and finishing timesassociated with a particular procedure. Additionally or alternatively,schedule 1430 may include a data structure configured to representinformation related to a schedule of at least one operating room and/orrelated to a schedule of at least one surgical procedure, such as ascheduled time associated with completion of the ongoing surgicalprocedure, as well as scheduled times for starting and finishing futuresurgical procedures.

FIG. 15 shows an example schedule 1430 that may include a listing ofprocedures such as procedures A C (e.g., surgical procedures, or anyother suitable medical procedures that may be performed in an operatingroom for which schedule 1430 is used). For each procedure A C, acorresponding starting and finishing times may be determined. Forexample, for a past procedure A, a starting time 1521A and a finishingtime 1521B may be the actual starting and finishing times. (Sinceprocedure A is completed, the schedule 1430 may be automatically updatedto reflect actual times). FIG. 15 shows that for a current procedure B,a starting time 1523A may be actual and a finishing time 1523B may beestimated (and recorded as an estimated time). Additionally, forprocedure C, that is scheduled to be performed in the future, a startingtime 1525A and a finishing time 1525B may be estimated and recorded. Itshould be noted that schedule 1430 is not limited to displaying and/orholding listings of procedures and starting/finishing times for theprocedures, but may include various other data associated with anexample surgical procedure. For example, schedule 1430 may be configuredto allow a user of schedule 1430 to interact with various elements ofschedule 1430 (for cases when schedule 1430 is represented by a computerbased interface such as a webpage, a software application, and/oranother interface). For example, a user may be allowed to click over orotherwise select areas 1513, 1515 or 1517 to obtain details forprocedures A, B or C respectively. Such details may include patientinformation (e.g., patient's name, age, medical history, etc.), surgicalprocedure information (e.g., a type of surgery, type of tools used forthe surgery, type of anesthesia used for the surgery, and/or othercharacteristics of a surgical procedure), and healthcare providerinformation (e.g., a name of a surgeon, a name of an anesthesiologist,an experience of the surgeon, a success rate of the surgeon, a surgeonrating based on surgical outcomes for the surgeon, and/or other datarelating to a surgeon). Some or all of the forgoing information mayalready appear in areas 1513, 1515 and 1517, without the need forfurther drill down.

In various embodiments, information for a surgical procedure may beentered by a healthcare provider (e.g., a nurse, a surgical assistant, asurgeon, and/or other healthcare professional) via an example form 1601,as shown in FIG. 16 . For example, form 1601 may have an “URGENCY”field, in which the healthcare provider may specify the urgency of thescheduled surgical procedure, a “SURGERY TYPE” field, in which thehealthcare provider may specify a type of the surgical procedure (or aname of the surgical procedure), a “Complications” field, in which thehealthcare provider may specify medical historical events for a patientthat may lead to complications during the surgical procedure, “PatientProfile” fields such as “Name”, “Address”, “Birthday”, “Contact”, and“Emergency Contact”, in which the healthcare provider may specify thecorresponding information about the patient. Further, form 1801 mayinclude a “Medical History” field that may be used to describe medicalhistory of a patient (e.g., “Medical History” field may be a pulldownlist, a space in which the healthcare provider may type text describingthe medical history for the patient, or any other suitable graphicaluser interface element that can be used for the description of themedical history for the patient. Additionally, form 1601 may include“Surgical Team” related fields that may specify names andresponsibilities of medical personnel who are scheduled to provide thesurgical procedure for the patient. Information about multiplehealthcare providers may be added by means of “Add Next Member” button,as shown in FIG. 16 . Form 1601 is only one illustrative example of aform with a few exemplary fields that can be used to input informationabout surgical procedures into schedule 1430, and any other suitableform may be used that allows for entering relevant information forschedule 1430. The number of fields of information on the form and thetype of information identified for capture may be a matter ofadministrator preference. Additionally or alternatively, information fora surgical procedure may be received from other sources, such as aHospital Information System (HIS), an Electronic Medical Record (EMR), aplanned operating room schedule, a digital calendar, an external system,and so forth.

Aspects of embodiments for enabling adjustments of an operating roomschedule may include accessing a data structure containing informationbased on historical surgical data and analyzing the visual data of theongoing surgical procedure and the historical surgical data to determinean estimated time of completion of the ongoing surgical procedure. Invarious embodiments, any steps of the method may be executed by one ormore processors of system 1410 executing software instructions 1416.

The data structure may be stored in database 1411 and may be accessedvia network 1418, or may be stored locally in a memory of system 1410.The data structure containing historical surgical data may include anysuitable data (e.g., image data, video data, text data, numerical data,spreadsheets, formulas, software codes, computer models, and/or otherdata objects), as well as any suitable relationships among various datavalues (or combinations of data values). The data may be storedlinearly, horizontally, hierarchically, relationally, non-relationally,uni dimensionally, multidimensionally, operationally, in an orderedmanner, in an unordered manner, in an object-oriented manner, in acentralized manner, in a decentralized manner, in a distributed manner,in a custom manner, or in any manner enabling data access. By way ofnon-limiting examples, data structures may include an array, anassociative array, a linked list, a binary tree, a balanced tree, aheap, a stack, a queue, a set, a hash table, a record, a tagged union.ER model, and a graph. For example, a data structure may include an XMLcode, an XML database, an RDBMS database, an SQL database or NoSQLalternatives for data storage/search such as, for example, MongoDB,Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk,Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. A datastructure may be a component of the disclosed system or a remotecomputing component (e.g., a cloud-based data structure). Data in thedata structure may be stored in contiguous or non contiguous memory.Moreover, a data structure, as used herein, does not require informationto be co-located. It may be distributed across multiple servers, forexample, that may be owned or operated by the same or differententities. Thus, the term “data structure” as used herein in the singularis inclusive of plural data structures.

In an example embodiment, the data structure may include a type ofprocedure (e.g., bypass surgery, bronchoscopy, or any other surgicalprocedure as described above), one or more characteristics of a patient(e.g., age, gender, medical considerations that may affect theprocedure, past medical history, and/or other patient information),name(s) and/or characteristics of operating surgeon and/oranesthesiologist, and a time that it took to complete the procedure. Insome cases, time for completion of the procedure may include a time forpreparing the operating room, a time for preparing a patient for thesurgical procedure, a time needed for medical personnel (i.e., nurses,surgeon, anesthesiologist, etc.) a time needed for the patient to beanesthetized or to fall asleep, a time needed for cleaning the operatingroom or any other surgery-related time needed to place the operatingroom in a condition for the next surgical procedure.

In an example embodiment, the data structure may be a relationaldatabase having one or more database tables. For instance, FIG. 17Aillustrates an example of data structure 1701 that may include datatables 1711 and 1713. In an example embodiment, data structure 1701 maybe part of relational databases, may be stored in memory, and so forth.Tables 1711 and 1713 may include multiple records (e.g., records 1 and2, as shown in FIG. 17A) and may have various fields, such as fields“Record Number”, “Procedure”, “Age”, “Gender”, “Medical Considerations”,“Time”, and “Other Data”. For instance, field “Record Number” mayinclude a label for a record that may be an integer, field “Procedure”may include a name of a surgical procedure, field “Age” may include anage of a patient, field “Gender” may include a gender of the patient,field “Medical Considerations” may include information about medicalhistory for the patient that may be relevant to the surgical procedurehaving the name as indicated in field “Procedure”, field “Time” mayinclude time that it took for the surgical procedure, and field “OtherData” may include links to any other suitable data related to thesurgical procedure. For example, as shown in FIG. 17A, 1711 may includelinks to data 1712A that may correspond to image data, data 1712B thatmay correspond to video data, data 1712C that may correspond to textdata (e.g., notes recorded during or after the surgical procedure,patient records, postoperative report, etc.), and data 1712D that maycorrespond to an audio data. In various embodiments, image, video, oraudio data may be captured during the surgical procedure. In some cases,video data may also include audio data. Image, video, text or audio data1712A-1712D are only some of the data that may be collected during thesurgical procedure. Other data may include vital sign data of thepatient, such as heart rate data, blood pressure data, blood test data,oxygen level, or any other patient-related data recorded during thesurgical procedure. Some additional examples of data may include roomtemperature, type of surgical instruments used, or any other datarelated to the surgical procedure and recorded before, during or afterthe surgical procedure.

As shown in FIG. 17A, tables 1711 and 1713 may include a record for asurgical procedure. For example, record 1 of table 1711 indicates that abypass surgical procedure was performed on a male of 65 years old,having a renal disease and that the bypass surgery was completed in 4hours. A record 2 of table 1711 indicates that a bypass surgicalprocedure was performed on a female of 78 years old, having nobackground medical condition that may complicate the surgical procedure,and that the bypass surgery was completed in 3 hours. Table 1713indicates that the bypass surgery for the male of 65 years old wasconducted by Dr. Mac, and that the bypass surgery for the female of 78years old was conducted by Dr. Doe. The patient characteristics such asage, gender, and medical considerations listed in table 1711 are onlysome of the example patient characteristics, and any other suitablecharacteristics may be used to differentiate one surgical procedure fromanother. For example, patient characteristics may further includepatient allergies, patient tolerance to anesthetics, various particularsof a patient (e.g., how many arteries need to be treated during thebypass surgery), a weight of the patient, a size of the patient,particulars of anatomy of the patient, or any other patient relatedcharacteristics which may have an impact on a duration (and success) ofthe surgical procedure.

Data structure 1701 may have any other number of suitable tables thatmay characterize any suitable aspects of the surgical procedure. Forexample, 1701 may include a table indicating an associatedanesthesiologist's identity, the time of day of the surgical procedure,whether the surgical procedure was a first, a second, a third, etc.procedure conducted by a surgeon (e.g., in the surgeon lifetime, withina particular day, etc.), an associated anesthesiologist nurse assistant,whether there were any complications during the surgical procedure, andany other information relevant to the procedure.

Accessing a data structure may include reading and/or writinginformation to the data structure. For example, reading and/or writingfrom/to the data structure may include reading and/or writing anysuitable historical surgical data such as historic visual data, historicaudio data, historic text data (e.g., notes during an example historicsurgical procedure), and/or other historical data formats. In an exampleembodiment, accessing the data structure may include reading and/orwriting data from/to database 111 or any other suitable electronicstorage repository. In some cases, writing data may include printingdata (e.g., printing reports containing historical data on paper).

Disclosed embodiments may further include analyzing the visual data ofthe ongoing surgical procedure using the data structure to determine anestimated completion time of the ongoing surgical procedure. Theestimated completion time may be any suitable indicator of estimatedcompletion of a surgical procedure, including, for example, a time ofday at which a surgical procedure is expected to complete, a timeremaining until completion, an estimated overall duration of thesurgical procedure, a probability distribution time values forcompletion of a surgical procedure, and so forth. Furthermore,completion time may include additional statistical informationindicating a likelihood of completion, based on historical surgical data(e.g., standard deviation associated with historical completion times,average historical completion times, mean for historical completiontimes, and/or other statistical metrics of completion times). In someexamples, a machine learning model may be trained using trainingexamples to estimate completion time of surgeries from images and/orvideos, and the trained machine learning model may be used to analyzethe visual data and determine the estimated completion time of theongoing surgical procedure. An example of such training example mayinclude an image and/or a video of a surgical procedure, together with alabel indicating the estimate completion time of the surgical procedure.For example, labels of the training examples may be based on at leastone of the data structure containing information based on historicalsurgical data, the historical data, user input, and so forth. Forexample, the training example may include images and/or videos from atleast one of the data structure containing information based onhistorical surgical data, the historical data, and so forth.

In one example, prior to starting the surgical procedure, the historicalsurgical data may be analyzed to determine an initial estimatedcompletion time of the ongoing surgical procedure (also herein referredto as a time of completion), or the initial estimated completion time ofthe ongoing surgical procedure may be received in other ways, forexample from a user, from a scheduling system, from an external system,and so forth.

In some embodiments, an average historical completion time may be usedto determine an estimated completion time. For example, the averagehistorical completion time may be calculated for historical surgicalprocedures that are of the same type as an ongoing surgical procedure,and the average historical completion time may be used as the estimatedcompletion time. In another example, similar historical surgicalprocedures may be selected (for example, using a k-Nearest Neighborsalgorithm, using a similarity measure between surgical procedures,etc.), and the average historical completion time may be calculated forthe selected similar historical surgical procedures.

The analysis of the historical data may involve any suitable statisticaldata analysis, such as determining an expected completion time valuebased on a probability distribution function, using Bayesianinterference, to determine how the probability distribution function isaffected by various patient/surgeon characteristics (e.g., an age of thepatient), linear regression, and/or other methods of quantifyingstatistical relationships. For instance, FIG. 17B shows an example graph1703 of points 1715 representing a distribution of completion time of aparticular surgical procedure (e.g., a bypass surgery) for patients ofdifferent ages. For example, a point 1715A shows that in a particularcase, for a patient of age A0, it took time T0 to complete the surgicalprocedure. Data for points 1715 may be used to construct a linearregression model 1717, and regression model 1717 may be used todetermine expected completion time T1 for a patient of age A1 accordingto point 1718 on the linear regression model. While graph 1703 shows thedependence of the completion time on one characteristic parameter of apatient (e.g., age of the patient), completion time may depend onmultiple characteristic parameters (e.g., the weight of a patient,characteristics of the healthcare professional conducting a surgicalprocedure, characteristics of an anesthesiologist, and other datadescribing a patient or procedure), as previously discussed, and points1715 may be plotted in a multi-dimensional Cartesian coordinate system,and regression model 1717 may include multivariate regression model. Inother examples, regression model 1717 may include a non-linearregression model.

In an example embodiment, determining the estimated completion time maybe based on one or more stored characteristics associated with ahealthcare professional conducting the ongoing surgical procedure. Suchcharacteristics may include age, a name, years of experience, alocation, of the healthcare professional, past performances, and/orother information describing a healthcare professional, for example, asdescribed above. The characteristics may be stored using any suitabledata structure using any suitable electronic (or in some cases, paper)storage. In an example embodiment, the characteristics may be stored ina database (e.g., database 1411, as shown in FIG. 14 ). For instance,based on an analysis of a historical data for a given healthcareprofessional for a given type of surgery, an expected completion timemay be estimated (e.g., the expected completion time may be an averagecompletion time determined from the historical data for a givenhealthcare professional for a given type of surgery). Furthermore, usinghistoric data for a given healthcare professional for a given type ofsurgery other statistics may be determined (e.g., standard deviationfrom the expected completion time, correlation of the expectedcompletion time with other characteristics of a surgical procedure, suchas an age of a patient or a time of the day the surgery is performed,and/or other statistic generated from historic completion times).

FIG. 18 shows an exemplary embodiment of obtaining a completion time1815 using a machine learning model 1813. Model 1813 may take as inputparameters 1811 various characteristics of a patient, variouscharacteristics of medical personnel, as well as a type of surgicalprocedure administered to the patient. For example, parameter P1, asshown in FIG. 18 , may indicate a type of surgical procedure, parameterP2 may indicate an age of a patient, parameter PN may indicate theweight of the patient, and the like. Various other parameters may beused, such as a type of surgical instrument being used, a size ofanatomical structure being operated on, and the like.

In various embodiments, completion time 1815 may be calculated usingmodel 1813 that may include machine learning models, such as neuralnetworks, decision trees, models based on ensemble methods (such asrandom forests), or any other machine learning model, for example asdescribed above. In some cases, model 1813 may be configured to return asingle number related to a completion time, and in some embodiments,model 1813 may be configured to return a probability distribution for acompletion time.

In various embodiments, model 1813 may be trained using a data setcontaining suitable parameters 1811 corresponding to historical surgicaldata that may include historical completion times for various patientsundergoing a given surgical procedure.

Embodiments of the disclosure may further include analyzing the visualdata of the ongoing surgical procedure and the historical surgical datato determine an estimated time of completion of the ongoing surgicalprocedure. Such analyzing may occur through machine learning and/orother techniques described herein for determining an estimatedcompletion time. In one example embodiment, to determine the completiontime for the surgery, the method may utilize a machine learning modelthat takes as an input information (such as a type of the surgicalprocedure, one or more of visual data of the ongoing surgical proceduresuch as images of the surgery or video data of the surgery, patientand/or medical personnel characteristics), and as an output returns anestimate of completion time. In some examples, the historical surgicaldata and the visual data of the ongoing surgical procedure may beanalyzed to identified records in the historical surgical data that aresimilar to the ongoing surgical procedure, for example using a visualsimilarity function, using an inexact graph matching algorithm on graphsrepresenting the visual data, using a k-Nearest Neighbors algorithm, andso forth. Further, in some examples, the identified records may be usedto determine the estimated time of completion of the ongoing surgicalprocedure. For example, a function (such as mean, median, mode,statistical function, linear function, non-linear function, etc.) of thetime of completion from the identified records may be calculated, theestimated time of completion of the ongoing surgical procedure may bebased on the calculated function. In an example embodiment, the visualdata of the ongoing surgical procedure may be collected at timesseparated by predetermined time intervals (e.g., the visual data may becollected every second, every few seconds, every few tens of seconds,every minute, or at any other appropriate interval. Additionally oralternatively, the visual data may be collected at times requested bymedical personnel (e.g., the visual data may be collected at timesrequested by a surgeon and/or anesthesiologist and/or a nurse, or anyother designated individual). For example, the surgeon may produce avisual/audio signal (e.g., a hand gesture, a body gesture, a visualsignal produced by a light source generated by a medical instrument, aspoken word, or any other trigger) that may be captured by one or moreimage sensors/audio sensors and recognized as a trigger for collectingthe visual data. Additionally or alternatively, the visual data may becollected based on a detected characteristic event during a surgicalprocedure, as further described below.

In various embodiments, adjusting an operating room schedule may includeusing historical visual data to train a machine learning model toestimate completion times, and wherein calculating the estimated time ofcompletion includes implementing the trained machine learning model. Anexample of input data for a machine learning model may include multiplevisual data records and parameters. A record of the visual data may be aset of images and/or multiple frames of a video captured by imagesensors for a particular time interval during the surgical procedure.For example, visual data record may be video data for the first fewminutes of the surgical procedure, the visual data record may be videodata for the next few minutes of the surgical procedure, and the visualdata record may be video data for the following few minutes of thesurgical procedure. In some examples, the machine learning model may betrained and/or used as described above.

Aspects of disclosed embodiments may include accessing a schedule forthe surgical operating room, including a scheduled time associated withcompletion of the ongoing surgical procedure. In an example embodiment,accessing may include reading and/or writing information to a schedule.One example of such schedule may include schedule 1430, or a datastructure containing information similar to the information described inrelation to schedule 1430. For example, reading and/or writing from/toschedule 1430 may include reading and/or writing any suitable datarelated to a past, present or future surgical procedure thatcorrespondingly was previously performed, or ongoing or scheduled to beperformed in the surgical operating room. Such data may include a nameof a procedure, a surgeon performing the procedure, a name of a patient,any characteristic parameters related to the patient or/and medicalpersonnel, a starting time (or an estimated starting time) for theprocedure and a finishing time (or an estimated finishing time) for theprocedure. In various embodiments, system 1410 may be used to readand/or write to schedule 1430.

Various embodiments may further include calculating, based on theestimated completion time of the ongoing surgical procedure, whether anexpected time of completion is likely to result in a variance from thescheduled time associated with the completion, and outputting anotification upon calculation of the variance, to thereby enablesubsequent users of the surgical operating room to adjust theirschedules accordingly. For example, the estimated (also referred to asexpected) time of completion of the ongoing surgical procedure may beobtained using any of the approaches discussed above (e.g., usingmachine learning models described above and/or linear regression modelsfor historical surgical data). The expected time of completion may becompared to an estimated finishing time for an example medical procedure(e.g., estimated finishing time 1523B, as shown in FIG. 15 ) and ifexpected time of completion does not substantially match time 1523B(e.g., expected time of completion is later than or prior to time1523B), the method may be configured to calculate a difference betweenthe expected time of completion and time 1523B. If the difference issmaller than a predetermined threshold value (e.g., the threshold valuemay be a minute, a few minutes, five minutes, ten minutes, fifteenminutes, and/or other time values), the method may determine that theexpected time of completion is substantially the same as time 1523B.Alternatively, if the difference is sufficiently large (i.e., largerthan a predetermined threshold value), the method may calculate (i.e.,determine) based on the estimated time of completion of the ongoingsurgical procedure that expected time of completion is likely to resultin a variance from the scheduled time associated with the completion. Invarious embodiments, the estimated completion time may be a duration oftime for completing a surgical procedure, and the expected time forcompletion may be an expected time at which the surgical procedure iscompleted.

In various embodiments, if the variance is detected, a notification maybe outputted upon determining the variance (e.g., the variance may bedetermined by calculating the difference between the expected time ofcompletion and time 1523B). In an example embodiment, the notificationmay include an updated operating room schedule. For example, updates toschedule 1430 may include text updates, graphics updates, or any othersuitable updates (e.g., video data, animations, or audio data).Additionally or alternatively, the notification may be implemented as awarning signal (e.g., light signal, audio signal, and/or other types oftransmission signals). In some cases, the notification may be an SMSmessage, an email, and/or other type of communication delivered to anysuitable devices (e.g., smartphones, laptops, pagers, desktops, TVs, andothers previously discussed) in possession of various users (e.g.,various medical personnel, administrators, patients, relatives orfriends of patients, and other interested individuals). For example, thenotification may be an electronic message transmitted to a device (asdescribed earlier) associated with a subsequent scheduled user (e.g., asurgeon, an anesthesiologist, and/or other healthcare professional) ofthe surgical operating room. Such notification may enable various users(e.g., users of the operating room) to adjust their schedules inaccordance with an update to the schedule. In various embodiments, theupdated operating room schedule may enable a queued healthcareprofessional to prepare for a subsequent surgical procedure. Forexample, if the expected time for completion of a surgical procedure ispast the estimated finishing time (e.g., time 1523B), a queuedhealthcare professional (e.g., a surgeon, an anesthesiologist, a nurse,etc.) may delay preparing for the surgical procedure. Alternatively, ifthe expected time for completion of a surgical procedure is prior totime 1523B), a queued healthcare professional (e.g., a surgeon, ananesthesiologist, a nurse, etc.) may start preparation for the surgicalprocedure at an earlier time than previously scheduled.

Aspects of disclosed embodiments may further include determining anextent of variance from a scheduled time associated with completion, inresponse to a first determined extent, outputting a notification, and inresponse to a second determined extent, forgoing outputting thenotification. For example, if the first determined extent is above apredetermined threshold value (e.g., above a few minutes, a few tens ofminutes, and/or other measure of time), some embodiments may determinethat such a first determined extent may influence scheduling time ofother surgical procedures. For such cases, a notification of thevariance may be transmitted to any suitable receiving party (e.g., tohealthcare providers administering a following surgical procedure).Alternatively, if it is determined that the second determined extent issufficiently small (e.g., smaller than a predetermined threshold value),embodiments may be configured not to transmit a notification.

Aspects of disclosed embodiments may further include determining whetheran expected completion time is likely to result in a delay of at least aselected threshold amount of time from a scheduled time associated withcompletion. In some embodiments, such determination may be made using asuitable machine learning model, such as model 1813 as described above.The selected threshold amount may be any suitable predetermined amount(e.g., a few minutes, a few tens of minutes, a half an hour, an hour,and/or other measure of time). For example, the selected thresholdamount may be based on operations of the surgical operating room.Additionally or alternatively, the selected threshold amount may bebased on a future event in a schedule for a surgical operating room. Forexample, if there is a second surgical procedure scheduled thirtyminutes after completion of a first surgical procedure, the selectedthreshold amount for the first surgical procedure may not exceed thethirty minutes. Additionally or alternatively, the selected thresholdamount of time may be selected based on subsequent users of the surgicaloperating room. For example, if a surgical procedure for subsequentusers may require substantial advanced preparation, the selectedthreshold amount may be sufficiently small (e.g., a few minutes).Alternatively, if the surgical procedure for subsequent users may notrequire substantial advanced preparation, and may be easily delayed orrescheduled, the selected threshold amount may be sufficiently large(e.g., thirty minutes, an hour, and/or other measure of time.) In somecases, urgency or importance of a surgical procedure for subsequentusers may determine a selected threshold amount. For example, for urgentsubsequent surgical procedures, an early notification may be needed,thus, requiring a short selected threshold amount.

In response to a determination that the expected time of completion islikely to result in a delay of at least the selected threshold amount oftime, disclosed embodiments may include outputting a notification. Asdescribed before, the notification may be any type of electronic orpaper data that may be output (such as by system 1410, as shown in FIG.14 ) for analyzing completion times. In an example embodiment, system1410 may be configured to output a notification as an electronic messageto a device of a healthcare provider, consistent with disclosedembodiments. In response to a determination that the expected time ofcompletion is not likely to result in a delay of at least the selectedthreshold amount of time, the method may be configured to forgooutputting the notification.

In some cases, disclosed embodiments may further include determiningwhether a surgical procedure is likely to conclude ahead of time (i.e.,an expected completion time for a surgical procedure is shorter than aplanned time for the surgical procedure). In response to a determinationthat the expected completion time is likely to be shorter than theplanned time for the surgical procedure by at least a selected thresholdamount of time, embodiments may be configured to output a notificationand/or forgo outputting the notification.

FIG. 19 shows an example process 1901 for adjusting an operating roomschedule consistent with disclosed embodiments. At step 1911, theprocess may include receiving visual data from an image sensor. Thevisual data may include image/video data tracking an ongoing surgicalprocedure. In an example embodiment, the visual data may be collected byvarious image sensors. In some cases, two or more image sensors (e.g.,cameras) may capture the visual data of the same region of the surgicalprocedure (e.g., a ROI) from different viewpoints. Additionally oralternatively, two or more image sensors may capture the visual data ofthe ROI using different magnifications. For example, a first imagesensor may capture an overview of the ROI, and a second image sensor maycapture an immediate area in the vicinity of a surgical tool locatedwithin the ROI.

At step 1913, process 1901 may include accessing a data structurecontaining historical surgical data as described above. At step 1915,process 1901 may include analyzing the visual data of the ongoingsurgical procedure and historical surgical data to determine anestimated time of completion of the ongoing surgical procedure. Aspreviously described, the analysis may use a statistical approach foranalyzing first historical surgical data (e.g., calculating the averageestimated time of completion for surgical procedures that are of thesame type as the ongoing surgical procedure and have similarcharacteristics as the ongoing surgical procedure). Additionally oralternatively, the analysis may involve training and using a machinelearning method for determining an estimated time of completion for anongoing surgical procedure. In some cases, several different analysisapproaches may be used, and estimated time of completion may bedetermined as an average time for times of completion obtained usingdifferent analysis approaches.

At step 1917, process 1901 may include accessing a schedule for thesurgical operating room using any suitable means. For example, accessingmay include accessing via a wired or wireless network via input devices(e.g., keyboard, mouse, etc.) or via any other means for allowingreading and/or writing data from/to the schedule.

At step 1919, process 1901 may include calculating whether the expectedtime of completion may result in a variance from the scheduled timeassociated with completion of surgical procedure, as described above. Ifthe variance is expected (step 1921, Yes), process 1901 may includeoutputting a notification at step 1923, as described above. Followingstep 1923, process 1901 may be completed. If the variance is notexpected (step 1921, No), process 1901 may be completed.

Aspects of the disclosed embodiments for enabling adjustments of anoperating room schedule may include analyzing the visual data, where aprocess of analyzing may include detecting a characteristic event in thereceived visual data, assessing the information based on historicalsurgical data to determine an expected time to complete the surgicalprocedure following an occurrence of the characteristic event inhistorical surgical data and determining the estimated time ofcompletion based on the determined expected time to complete. Forexample, the characteristic event may be detected in the received visualdata, as described above. In some examples, the historical surgical datamay include a data structure connecting characteristic events withexpected time to complete a surgical procedure. For example, thehistorical surgical data may include a data structure that specifies afirst time to complete a surgical procedure from a first event, and asecond time to complete a surgical procedure from a second event, thesecond time may differ from the first time. Further, the data structuremay be accessed using the detected characteristic event to determine thetime to complete the surgical procedure from the occurrence of thecharacteristic event.

In various embodiments, a detected characteristic event in the receivedvisual data may refer to a particular procedure or action performed by amedical professional (e.g., by a surgeon, by an anesthesiologist, nurse,and/or other medical professional). For example, characteristic eventsof a laparoscopic cholecystectomy surgery may include trocar placement,calot's triangle dissection, dipping and cutting of cystic duct andartery, gallbladder dissection, gallbladder packaging, cleaning andcoagulation of liver bed, gallbladder retraction, and so forth. Inanother example, surgical characteristic events of a cataract surgerymay include povidone-iodine injection, corneal incision, capsulorhexis,phaco-emulsification, cortical aspiration, intraocularlens implantation,intraocular-lens adjustment, wound sealing, and so forth. In yet anotherexample, surgical characteristic events of a pituitary surgery mayinclude preparation, nasal incision, nose retractor installation, accessto the tumor, tumor removal, column of nose replacement, suturing, nosecompress installation, and so forth. Some other examples of surgicalcharacteristic events may include incisions, laparoscope positioning,suturing, and so forth. In this context, characteristic event mayinclude any event commonly occurring within a particular stage of asurgical procedure, any event commonly suggesting a particularcomplication within a surgical procedure, or any event commonlyoccurring in response to a particular complication within a surgicalprocedure. Some non-limiting examples of such characteristic events mayinclude usage of particular medical tools, performance of particularactions, infusion of a particular substance, call to a particularspecialist, order of a particular device, instrument, equipmentmedication, blood, blood products, or supply, particular physiologicalresponse, and so forth.

A characteristic event (also referred to as an intraoperative surgicalevent) may be any event or action that occurs during a surgicalprocedure or phase. In some embodiments, an intraoperative surgicalevent may include an action that is performed as part of a surgicalprocedure, such as an action performed by a surgeon, a surgicaltechnician, a nurse, a physician's assistant, an anesthesiologist, adoctor, or any other healthcare professional. The intraoperativesurgical event may be a planned event, such as an incision,administration of a drug, usage of a surgical instrument, an excision, aresection, a ligation, a graft, suturing, stitching, or any otherplanned event associated with a surgical procedure or phase. In someembodiments, the intraoperative surgical event may include an adverseevent or a complication. Some examples of intraoperative adverse eventsmay include bleeding, mesenteric emphysema, injury, conversion tounplanned open surgery (for example, abdominal wall incision), incisionsignificantly larger than planned, and so forth. Some examples ofintraoperative complications may include hypertension, hypotension,bradycardia, hypoxemia, adhesions, hernias, atypical anatomy, duraltears, periorator injury, arterial occlusions, and so forth. Theintraoperative event may include other errors, including technicalerrors, communication errors, management errors, judgment errors,decision-making errors, errors related to medical equipment utilization,miscommunication, or any other mistakes.

In various embodiments, events may be short or may last for a durationof time. For example, a short event (e.g., incision) may be determinedto occur at a particular time during the surgical procedure, and anextended event (e.g., bleeding) may be determined to occur over a timespan. In some cases, extended events may include a well definedbeginning event and a well defined ending event (e.g., beginning ofsuturing and ending of the suturing), with suturing being an extendedevent. In some cases, extended events are also referred to as phasesduring a surgical procedure.

A process of assessing information based on historical surgical data todetermine an expected time to complete a surgical procedure following anoccurrence of a characteristic event in historical surgical data mayinvolve using a suitable statistical approach for analyzing completiontimes of historical surgical procedures that include the occurrence ofthe characteristic event. For example, the completion times may beanalyzed to determine an average completion time for such procedures,and the average completion time may be used as the expected time tocomplete the surgical procedure. In Some embodiments may includedetermining an estimated time of completion (i.e., a time at which anexample surgical procedure containing a characteristic event will becompleted) based on the determined expected time to complete (i.e., theduration of time needed to complete the surgical procedure).

Embodiments for adjusting an operating room schedule may further includeusing historical visual data to train a machine learning model to detectcharacteristic events. In various embodiments, the machine learningmodel for recognizing a feature (or multiple features) may be trainedvia any suitable approach, such as, for example, a supervised learningapproach. For instance, historic visual data containing featurescorresponding to a characteristic event may be presented as input datafor the machine learning model, and the machine learning model mayoutput the name of a characteristic event corresponding to the featureswithin the historic visual data.

In various embodiments, detecting the characteristic event includesimplementing the trained machine learning model. The trained machinelearning model may be an image recognition model for recognizing afeature (or multiple features) within the visual data that may be usedas a trigger (or triggers) for the characteristic event. The machinelearning model may recognize features within one or more images orwithin a video. For example, features may be recognized within a videoin order to detect a motion and/or other changes between frames of thevideo. In some embodiments, image analysis may include object detectionalgorithms, such as Viola-Jones object detection, convolutional neuralnetworks (CNN), or any other forms of object detection algorithms. Otherexample algorithms may include video tracking algorithms, motiondetection algorithms, feature detection algorithms, color-baseddetection algorithms, texture based detection algorithms, shape-baseddetection algorithms, boosting based detection algorithms, facedetection algorithms, or any other suitable algorithm for analyzingvideo frames.

In some cases, characteristic events may be classified as positive(i.e., events that lead to positive outcomes) and adverse (i.e., eventsthat lead to negative outcomes). The positive outcomes and the negativeoutcomes may have different effect on the estimated completion time.

In some cases, the image recognition model may be configured not onlyrecognize features within the visual data but also configured to formconclusions about various aspects of the ongoing (or historical)surgical procedure based on analysis of the visual data (or historicalvisual data). For example, by analyzing visual data of an examplesurgical procedure, the image recognition model may be configured todetermine a skill level of a surgeon, or determine a measure of successof the surgical procedure. For example, if there are no adverse eventsdetermined in the visual data, the image recognition model may assign ahigh success level for the surgical procedure and update (e.g.,increase) the skill level of the surgeon. Alternatively, if adverseevents are determined in the visual data, the image recognition modelmay assign a low success level for the surgical procedure and update(e.g., decrease) the skill level of the surgeon. The algorithm forassigning success level for the surgical procedure and the process ofupdating the skill level of the surgeon may be determined based onmultiple factors such as the type of adverse events detected during anexample surgical procedure, the likelihood of an adverse event duringthe surgical procedure, given specific characteristics of a patient(e.g., patient age), the average number of adverse events for historicalsurgical procedures of the same type conducted for patients havingsimilar patient characteristics, the standard deviation from the averagenumber of adverse events for historical surgical procedures of the sametype conducted for patients having similar patient characteristics,and/or other metrics of adverse events.

In some cases, a process of analyzing visual data may includedetermining a skill level of a surgeon in the visual data, as discussedabove. In some cases, calculating the estimated time of completion maybe based on the determined skill level. For example, for each determinedskill level for a surgical procedure, an estimated time of completionmay be determined. In an example embodiment, such an estimated time ofcompletion may be based on historical times of completion correspondingto historical surgical procedures performed by surgeons with thedetermined skill level. For example, average historical times ofcompletion calculated for above-referenced historical times ofcompletion may be used to determine the estimated time of completion.Such an estimated time of completion may be stored in a database and maybe retrieved from the database based on a determined skill level.

Detecting a characteristic event using a machine learning method may beone possible approach. Additionally or alternatively, the characteristicevent may be detected in the visual data received from image sensorsusing various other approaches. In one embodiment, the characteristicevent may be identified by a medical professional (e.g., a surgeon)during the surgical procedure. For example, surgeon may identify thecharacteristic event using a visual or an audio signal from the surgeon(e.g., a hand gesture, a body gesture, a visual signal produced by alight source generated by a medical instrument, a spoken word, or anyother signal) that may be captured by one or more image sensors/audiosensors and recognized as a trigger for the characteristic event.

In various embodiments, enabling adjustments of an operating roomschedule may include analyzing historical times to complete the surgicalprocedure following an occurrence of the characteristic event inhistorical visual data. For example, embodiments may include computingaverage historical time to complete the surgical procedure (alsoreferred herein as an average historical completion time) following theoccurrence of the characteristic event in the historical visual data,and using the average historical completion time as an estimate for thecompletion time of the ongoing surgical procedure. In some cases,however, the estimated completion time may be calculated using otherapproaches discussed above (e.g., using machine learning methods), andthe average historical completion time may be updated based on thedetermined actual time to complete the ongoing surgical procedure (asdetermined after the completion of the ongoing procedure). In variousembodiments, the average historical completion time may be first updatedusing an estimated completion time, and then the update may be finalizedafter completion of the surgical procedure.

Additionally or alternatively, analyzing historical completion timesfollowing an occurrence of the characteristic event in order to estimatethe completion time may include using a machine learning model. Themachine learning model may be trained using a training examples toestimate completion time after occurrences of events, and the trainedmachine learning model may be used to estimate the completion time basedon the occurrence of the characteristic event. An example of suchtraining example may include an indication of a characteristic eventtogether with a label indicating the desired estimation of thecompletion time. In one example, a training example may be based onhistorical surgical data, for example representing an actual time tocompletion in an historical surgical procedure after the occurrence ofthe characteristic event in the historical surgical procedure. Inanother example, a training example may be based on user input, may bereceived from an external system, and so forth. The machine learningmodel may also be trained to base the estimation of the completion timeon other input parameters, such as various characteristics of a patient,various characteristics of a medical personnel, as well as a type ofsurgical procedure administered to the patient (e.g., parameters 1811,as shown in FIG. 18 ) as well as one or more characteristic eventsduring the surgical procedure. Further, such input parameters may beprovided to the trained machine learning model to estimate thecompletion time.

As described before, embodiments of the present disclosure may include asystem, process, or computer readable media for analyzing the visualdata of the ongoing surgical procedure and the historical surgical datato determine an estimated time of completion of the ongoing surgicalprocedure. In an example embodiment, analyzing may include determiningthe estimated time of completion based on the analysis of the historicaltimes. The estimate for the completion time may be determined using anysuitable approaches such as using a machine learning method (asdescribed above), or by computing an average historical time to completethe surgical procedure, and using such average historical time as theestimated completion time.

Aspects of embodiments for enabling adjustments of an operating roomschedule may further include detecting a medical tool in the visual dataand calculating the estimated completion time based on the detectedmedical tool. The medical tool (also referred to as a surgical tool) maybe one of the characteristic parameters of the surgery, such asparameters P1-PN, as shown in FIG. 18 that may affect a calculation ofthe estimated time of completion of the surgical procedure. As discussedabove, in an example embodiment, a machine learning method may be usedto calculate the estimated completion time based on various parametersP1-PN, such as, for example, a type of medical tool used during thesurgical procedure. Furthermore, detection of the medical tool in thevisual data tracking the ongoing surgical procedure may be achievedusing any suitable approach (e.g., using a suitable image recognitionalgorithm as described above). In one example, in response to adetection of a first medical tool, a first completion time may beestimated, and in response to a detection of a second medical tool, asecond completion time may be estimated, the second completion time maydiffer from the first completion time. In one example, in response to adetection of a first medical tool, a first completion time may beestimated, and in response to a detection of no medical tool, a secondcompletion time may be estimated, the second completion time may differfrom the first completion time.

In some cases, embodiments for analyzing visual data may also includedetecting an anatomical structure in the visual data and calculating theestimated time of completion based on the detected anatomical structure.The anatomical structure may be detected and identified in the visualdata using an image recognition algorithm. Additionally oralternatively, the anatomical structure may be identified by ahealthcare professional during an ongoing surgical procedure (e.g., thehealthcare professional can use gestures, sounds, words, and/or othersignals) to identify an anatomical structure. The visual data of theongoing surgical procedure depicting the anatomical structure may beused to calculate the estimated completion time. For example, suchvisual data may be used as an input to a machine learning method toobtain estimated completion time. In one example, in response to adetection of a first anatomical structure, a first completion time maybe estimated, and in response to a detection of a second anatomicalstructure, a second completion time may be estimated, the secondcompletion time may differ from the first completion time. In oneexample, in response to a detection of a first anatomical structure, afirst completion time may be estimated, and in response to a detectionof no anatomical structure, a second completion time may be estimated,the second completion time may differ from the first completion time.

Aspects of embodiments for analyzing visual data may include detectingan interaction between an anatomical structure and a medical tool in thevisual data and calculating the estimated time of completion based onthe detected interaction. For example, the interaction between ananatomical structure and a medical tool may be detected as describedabove. The interaction may include any action by the medical tool thatmay influence the anatomical structure or vice versa. For example, theinteraction may include a contact between the medical tool and theanatomical structure, an action by the medical tool on the anatomicalstructure (such as cutting, clamping, grasping, applying pressure,scraping, etc.), a physiological response by the anatomical structure,the medical tool emitting light towards the anatomical structure (e.g.,medical tool may be a laser that emits light towards the anatomicalstructure), a sound emitted towards anatomical structure, anelectromagnetic field created in a proximity of the anatomicalstructure, a current induced into an anatomical structure, or any othersuitable forms of interaction. In one example, in response to adetection of a first interaction between an anatomical structure and amedical tool, a first completion time may be estimated, and in responseto a detection of a second interaction between an anatomical structureand a medical tool, a second completion time may be estimated, thesecond completion time may differ from the first completion time. In oneexample, in response to a detection of a first interaction between ananatomical structure and a medical tool, a first completion time may beestimated, and in response to a detection of no interaction between ananatomical structure and a medical tool, a second completion time may beestimated, the second completion time may differ from the firstcompletion time.

The visual data of the ongoing surgical procedure depicting theanatomical structure and the medical tool may be used to calculate theestimated completion time. For example, such visual data may be used asan input to a machine learning method to obtain estimated completiontime, for example, as described above.

As previously discussed, the present disclosure relates to methods andsystems for enabling adjustments of an operating room schedule, as wellas non-transitory computer-readable medium that may include instructionsthat, when executed by at least one processor, cause the at least oneprocessor to execute operations enabling adjustment of an operating roomschedule and may include various steps of the method for enablingadjustments of an operating room schedule as described above.

Disclosed systems and methods may involve analyzing surgical footage toidentify features of surgery, patient conditions, and other features todetermine insurance reimbursement. Insurance reimbursement may need tobe determined for various steps of a surgical procedure. Steps of asurgical procedure may need to be identified, and insurancereimbursement codes may need to be associated with the identified steps.Therefore, there is a need for identifying steps of a surgical procedureusing information obtained from surgical footage and associatinginsurance reimbursement with these steps.

Aspects of this disclosure may relate to methods, systems, devices, andcomputer readable media for analyzing surgical images to determineinsurance reimbursement. For ease of discussion, a method is describedbelow, with the understanding that aspects of the method apply equallyto systems, devices, and computer readable media. For example, someaspects of such a method may occur electronically over a network that iseither wired, wireless, or both. Other aspects of such a method mayoccur using non-electronic means. In the broadest sense, the method isnot limited to particular physical and/or electronic instrumentalities,but rather may be accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method for analyzing surgicalimages to determine insurance reimbursement may include accessing videoframes captured during a surgical procedure on a patient. Embodimentsfor analyzing surgical images may include using any suitable approach(e.g., using a machine-learning approach) for determining phases ofsurgical procedure, events during a surgical procedure, anatomicalstructures being operated on, surgical instruments used during asurgical procedure, interactions of surgical instruments and anatomicalstructures, motion of surgical instruments, motion of anatomicalstructures, deformation of anatomical structures, color changes ofanatomical structures, leakage (e.g., bleeding) of anatomicalstructures, incisions within anatomical structures, or any other changesto anatomical structures (e.g., a rupture of an anatomical structure)during an example surgical procedure.

In various embodiments, insurance reimbursement may include informationregarding how much money may be paid by an insurance company and/or aninsurance program (such as a government health insurance program) for agiven surgical procedure or segments (portions) thereof. For example,insurance reimbursement may cover costs associated with all, or some ofthe segments of a surgical procedure. A segment of the surgicalprocedure may correspond to a segment of surgical footage of thesurgical procedure. In some cases, insurance reimbursement may cover anentire cost associated with a segment of a surgical procedure, and inother cases, the insurance reimbursement may partially cover a costassociated with a segment of a surgical procedure. Depending on a typeof surgical procedure (e.g., if the surgical procedure is elective for apatient), the insurance reimbursement may not cover costs associatedwith a segment (or an entirety) of a surgical procedure. In otherexamples, different reimbursement means (e.g., different reimbursementcodes) may exist for different patients and/or different surgicalprocedures (or for different actions associated with the surgicalprocedures) based on a condition of the patient and/or on properties ofthe surgical procedures.

In some embodiments, accessing video frames captured during a surgicalprocedure may include accessing a database (e.g., database 1411, asshown in FIG. 14 ) by a suitable computer-based software application.For example, a database may be configured to store video frames capturedduring various surgical procedures and may be configured to store anyother information related to a surgical procedure (e.g., notes fromsurgeons conducting a surgical procedure, vital signals collected duringa surgical procedure). As described herein, the surgical procedure mayinclude any medical procedure associated with or involving manual oroperative activities performed on a patient's body.

Consistent with disclosed embodiments, analyzing video frames capturedduring a surgical procedure to identify in the video frames at least onemedical instrument, at least one anatomical structure, and at least oneinteraction between the at least one medical instrument and the at leastone anatomical structure, for example as described above. In variousembodiments, analyzing video frames captured during a surgical proceduremay include using image recognition, as discussed herein. When analyzingsurgical footage, at least some frames may capture an anatomicalstructure (herein, also referred to as a biological structure). Suchportions of surgical footage may include one or more medical instruments(as described herein) interacting with one or more anatomicalstructures.

A medical instrument and an anatomical structure may be recognized insurgical footage using image recognition, as described in thisdisclosure and consisted with various disclosed embodiments. Aninteraction between a medical instrument and an anatomical structure mayinclude any action by the medical instrument that may influence theanatomical structure or vice versa. For example, the interaction mayinclude a contact between the medical instrument and the anatomicalstructure, an action by the medical instrument on the anatomicalstructure (such as cutting, clamping, grasping, applying pressure,scraping, etc.), a physiological response by the anatomical structure,the medical instrument emitting light towards the anatomical structure(e.g., surgical tool may be a light-emitting laser) a sound emittedtowards anatomical structure, an electromagnetic field in proximity tothe anatomical structure, a current induced into the anatomicalstructure, or any other form of interaction.

In some cases, detecting an interaction may include identifyingproximity of the medical instrument to an anatomical structure. Forexample, by analyzing the surgical video footage, a distance between themedical instrument and a point (or a set of points) of an anatomicalstructure may be determined through image recognition techniques, asdescribed herein.

Aspects of disclosed embodiments may further include accessing adatabase of reimbursement codes correlated to medical instruments,anatomical structures, and interactions between medical instruments andanatomical structures. By way of example, a correlation of areimbursement code with one or more medical instruments, one or moreanatomical structures and one or more interactions between medicalinstruments and anatomical structures may be represented in a datastructure such as one or more tables, linked lists. XML data, and/orother forms of formatted and/or stored data. In some embodiments, acorrelation may be established by a code-generating machine-learningmodel. In various cases, the reimbursement codes together withinformation on how the codes are correlated with medical instruments,anatomical structures and interactions between medical instruments andanatomical structures may be stored in a data structure.

FIG. 20 shows an example of data structure 2001 for providinginformation on how reimbursement codes are correlated with medicalinstruments, anatomical structures, and interactions between medicalinstruments. For example, data structure 2001 may include several tablessuch as tables 2011, 2013 and 2015. In various embodiments, an exampletable may include records (e.g., rows) and fields (e.g., columns). Forexample, table 2011 may have a field entitled “Record” containing recordlabels (e.g., “1”, as shown in FIG. 20 ). For each record, a fieldentitled “Code” may contain a reimbursement code (e.g., a code“1.20:11.30.50”), a field entitled “Procedure Segment” may contain anumber and possibly a name of a segment of a surgical procedure (e.g.,“1, Incision, Bypass Surgery”), a field entitled “1st Instrument” maycontain a number and possibly a name of a first medical instrument usedduring the segment of the surgical procedure (e.g., “20, Scalpel”), afield entitled “2nd Instrument” may contain a number and possibly a nameof a second medical instrument used during the segment of the surgicalprocedure (if such an instrument was present) (e.g., “11, Forceps”), afield entitled “Other Data” may contain any related data that may beused further to characterize the surgical procedure or segment thereof(e.g., such data may include a duration of the segment of the surgicalprocedure, a sequence of events during the segment of the surgicalprocedure, a sequence of instruments used during the surgical procedure(e.g., “Scalpel->Forceps” may indicate that scalpel was used beforeforceps), and/or other characteristics of the segment. An example table2013 may contain other related fields such as a field entitled “1stAnatomical Structure” that may contain a number and possibly a name ofan” anatomical structure (e.g., “30, Internal Mammary Artery”),associated with record “1”, as labeled in a field entitled “Record” intable 2013. Further, an example table 2015 may include field entitled“Record” for identifying the record, and a field “Interaction” that maycontain a description of an interaction between a medical instrument andan anatomical structure that may be represented by a number and possiblya name (e.g., “50, Incision of the Left Internal Mammary Artery”).Further, table 2015 may include a field entitled “Interaction Data” thatmay include links to image data 2012A, video data 2012B, text data2012C, and/or audio data 2012D, as shown in table 2015.

In various embodiments, reimbursement codes may have an internal datastructure, as shown by structure 2020. For example, a first number forreimbursement code may be a number associated with a segment of asurgical procedure (e.g., number “1”), a second set of numbers may beassociated with surgical instruments used during the segment of thesurgical procedure (e.g., numbers “20:11” may be associated with thefirst instrument labeled “20” and the second instrument labeled “11”), athird set of numbers may associated with anatomical structures beingoperated (e.g., “30”), and a fourth set of numbers may be associatedwith interactions of instruments and anatomical structures (e.g., “50”).In a different example, reimbursement code may be set by the insuranceprogram or by a regulator. In some examples, a single reimbursement codemay be associated with the entire surgical procedure.

Using a data structure to determine reimbursement codes based on medicalinstruments, anatomical structures, and interactions of medicalinstruments and anatomical structures may be one possible approach.Additionally, a code-generating machine-learning method may be used todetermine a reimbursement code for a surgical procedure or a segmentthereof. For example, a code-generating machine-learning method may takeas an input a segment of surgical footage and output a reimbursementcode for a segment of a surgical procedure represented by the segment ofthe surgical footage. In various embodiments, a code-generatingmachine-learning method may be a collection of various machine-learningmethods configured for various tasks. For example, the code-generatingmachine-learning method may include a first image recognition algorithmfor recognizing a medical instrument in a segment of surgical footageand a second image recognition algorithm for recognizing anatomicalstructures in a segment of the surgical footage. In various embodiments,image recognition algorithms may be any suitable algorithms (e.g.,neural networks), as described herein and consistent with variousdisclosed embodiments.

Disclosed embodiments may further include comparing an identified atleast one interaction between at least one medical instrument and atleast one anatomical structure with information in the database ofreimbursement codes to determine at least one reimbursement codeassociated with the surgical procedure. For example, embodiments mayinclude comparing an identified interaction with various details aboutinteractions stored in a database. Thus, by way of example, amachine-learning model (e.g., an image recognition algorithm) may beconfigured to identify an interaction within surgical footage and toclassify the interaction (e.g., an interaction may be classified byassigning a name to the interaction or determining a type of theinteraction). For example, a name or a type of an interaction may be“incision of the left internal mammary artery.” In some embodiments, amachine-learning model may be configured to analyze surgical footage andselect the most appropriate interaction from a list of possibleinteractions. Once the interaction is identified, the name (or otheridentification for the interaction) may be compared with anidentification of interactions stored in a database, and the databasemay be used to find a reimbursement code corresponding to the identifiedinteraction, or to a surgical procedure that includes the identifiedinteraction.

Identifying interactions using a machine-learning algorithm is onepossible approach. Additionally or alternatively, interactions may beidentified by a surgeon administering a surgical procedure, a nursepractitioner present during the surgical procedure, and/or otherhealthcare professionals. For example, an interaction may be identifiedby selecting a segment of surgical footage corresponding to theinteraction and assigning a name that may tag a segment. In variousembodiments, a computer-based software application may be used to dovarious manipulations with segments of surgical footage (such asassigning name tags to different segments, selecting different segments,and/or other data operations). The computer-based software applicationmay be configured to store related data (e.g., name tags for differentsegments of surgical footage, and starting and finishing time forsegments of surgical footage) in a database.

Various embodiments may further include outputting at least onereimbursement code for use in obtaining insurance reimbursement for thesurgical procedure. For example, a code-generating machine-learningmodel may be used to output at least one reimbursement code, asdescribed above. Alternatively, the reimbursement code may be output viaa query to a database containing reimbursement codes corresponding tointeractions of medical instruments with anatomical structures.

In some cases, outputting the reimbursement code may includetransmitting the reimbursement code to an insurance provider using anysuitable transmission approaches consistent with disclosed embodimentsand discussed herein.

In some cases, at least one reimbursement code outputted includes aplurality of outputted reimbursement codes. For example, multiplereimbursement codes may correspond to one or more segments of a surgicalprocedure. In one embodiment, the first reimbursement code mightcorrespond to an incision-related segment, and a second reimbursementcode may, for example, correspond to suturing-related segment. In somecases, multiple reimbursement codes may correspond to multiple medicalinstruments used to perform one or more operative actions during asegment of a surgical procedure. When more than one surgeon (or anyother healthcare professional) is present during a surgical procedure,multiple reimbursement codes may be determined for a procedure performedby each surgeon. And when more than one reimbursable procedure isperformed in a single segment, more than one reimbursement code may beoutput for that single segment.

In an example embodiment, at least two of the plurality of outputtedreimbursement codes may be based on differing interactions with a commonanatomical structure. For example, the first interaction may include afirst medical instrument interacting with an anatomical structure, and asecond interaction may include a second medical instrument interactingwith the anatomical structure. In some cases, the same instrument may beused for different types of interactions with an anatomical structure(e.g., forceps may be used to interact with an anatomical structure indifferent ways).

In some embodiments, at least two outputted reimbursement codes aredetermined based in part on detection of two different medicalinstruments. For example, a first and a second medical instrument may bedetected in surgical footage using any suitable method (e.g., using asuitable machine-learning approach or using information from ahealthcare provider). Both the first and the second medical instrumentmay be used at the same time, and in some cases, a second medicalinstrument may be used after using the first medical instrument. The useof a first medical instrument may partially overlap (in time) with theuse of a second medical instrument. In such instances, two or morereimbursement codes may be outputted, regardless of whether the medicalinstruments that triggered the codes were being used at the same time orat differing times.

In various embodiments determining at least one reimbursement code maybe based on an analysis of a post-operative surgical report. Forexample, to determine the reimbursement code for a particular segment ofa surgical procedure, a post-operative surgical report may be consultedto obtain information about the segment of the surgical procedure. Anyinformation related to a segment of a surgical procedure, and/or theinformation obtained from the post-operative report, may be used todetermine the reimbursement codes (e.g., events that occurred during asegment of a surgical procedure, surgical instruments used, anatomicalstructures operated upon, interactions of surgical instruments andanatomical structures, imaging performed, various measurementsperformed, number of surgeons involved, and/or other surgical actions).

In various embodiments, video frames of surgical footage may be capturedfrom an image sensor positioned above the patient, as described hereinand consistent with various described embodiments. For example, imagesensors 115, 121, 123, and/or 125, as described above in connection withFIG. 1 may be used to capture video frames of surgical footage. Inaddition, or alternatively, video frames may be captured from an imagesensor associated with a medical device, as described herein andconsistent with various described embodiments. FIG. 3 shows one exampleof a medical device having associated image sensors, as describedherein.

Embodiments for analyzing surgical images to determine insurancereimbursement may include updating a database by associating at leastone reimbursement code with the surgical procedure. The database may beupdated using any suitable means (e.g., using a machine-learning model,by sending appropriate data to the database, through SQL commands, bywriting information to memory, and so forth). For example, surgicalfootage of a surgical procedure may be analyzed, as described above, todetermine various segments of the surgical procedure for whichreimbursement codes may be associated. Once the reimbursement codes aredetermined, the codes may be associated with the surgical procedure andbe configured for storage in the data structure. The data structure mayassume any form or structure so long as it is capable or retaining data.By way of one example, the data structure may be a relational databaseand include tables with table fields storing information about thesurgical procedure (e.g., an example table field may include a name ofthe surgical procedure) and storing reimbursement codes associated withthe surgical procedure.

Various embodiments may include generating correlations betweenprocessed reimbursement codes and at least one of a plurality of medicalinstruments in historical video footage, a plurality of anatomicalstructures in the historical video footage, or a plurality ofinteractions between medical instruments and anatomical structures inthe historical video footage; and updating the database based on thegenerated correlations. In an exemplary embodiment, correlations may begenerated using any suitable means such as using machine-learningmethods and/or using an input of healthcare professionals, healthcareadministrators and/or other users. Correlations may be represented bytables (e.g., tables 2011-2015, as shown in FIG. 20 ), as describedabove. In some cases, the correlations may be generated for processedreimbursement codes (e.g., reimbursement codes relating to portions ofhistorical surgical procedures, for which a health insurer of a patienthas previously reimbursed a healthcare provider). For example,historical surgical data (e.g., historical surgical footage) may beanalyzed (e.g., using a machine-learning method) to determine one ormore medical instruments in historical video footage, one or moreanatomical structures in the historical video footage, or one or moreinteractions between medical instruments and anatomical structures inthe historical video footage. Provided that segments of historicalsurgical procedure have associated processed reimbursement codes (e.g.,the processed reimbursement codes were assigned to the segments of thehistorical surgical procedure using any suitable approach available inthe past, such as inputs from a healthcare provider), the processedreimbursement codes may be correlated with information obtained from thehistorical surgical data (e.g., information about medical instruments,anatomical structures, and interactions between medical instruments andanatomical structures identified in the historical surgical data).

In various embodiments, a machine-learning method for generatingcorrelations may be trained, as discussed in this disclosure. Historicalsurgical data may be used as part of the training process. For example,historical surgical footage for a given segment of a surgical proceduremay be provided as a machine-learning input, which thereafter determinesa reimbursement code. A reimbursement code may be compared with aprocessed reimbursement code for the given segment of the surgicalprocedure to determine if the machine-learning model outputs a correctprediction. Various parameters of the machine-learning model may bemodified using, for example, a backpropagation training process.

In various embodiments, as discussed herein, historical video frames maybe used to train any suitable machine learning model for various tasksbased on information contained within the video frames (i.e., anysuitable image-based information). As previously discussed,machine-learning models may detect at least one of medical tools,anatomical structures, or interactions between medical tools andanatomical structures. Once the model recognizes correlations, thosecorrelations can then be extrapolated to current video under analysis.

In some cases, generating correlations may include implementing astatistical model. For example, historical processed reimbursement codesmay be analyzed for similar segments of historical surgical proceduresto determine a correlation. A correlation may be between a reimbursementcode and various aspects of a segment of a surgical procedure. Surgicalsegments can be characterized by medical instruments, anatomicalstructures, and interactions between medical instruments and anatomicalstructures. If different processed reimbursement codes were used forsuch similar segments, then correlations may be generated by evaluatingthe most likely reimbursement code that should be used. For example, iffor a segment of a historical procedure of a given type, a processedreimbursement code C1 was used 100 times, a processed reimbursement codeC2 was used 20 times, and a processed reimbursement code C3 was used 10time, then reimbursement code C1 may be selected as the most likelyreimbursement code that should be used.

In some cases, when processed reimbursement codes are different for thesame (or similar) segments of historical surgical procedures,characteristics of these segments may be analyzed to determine if somedifferences in the characteristics of these segments may be responsiblefor a difference in processed reimbursement codes. In variousembodiments, differences in characteristics of segments of historicalsurgical procedures may correlate the difference in processedreimbursement codes (as measured using any suitable statisticalapproach).

In various embodiments, after generating the correlations, as describedabove, a database may be updated based on the generated correlations.For example, for a given medical instrument interacting with a givenanatomical structure, an expected reimbursement code (or, in some cases,a set of possible reimbursement codes) may be associated and stored inthe database. A set of possible reimbursement codes may be used tofurther narrow a particular one of the reimbursement codes based oncharacteristics associated with a segment of a surgical procedureidentified in surgical footage.

Additionally or alternatively, disclosed embodiments may includereceiving a processed reimbursement code associated with a surgicalprocedure and updating the database based on the processed reimbursementcode. The processed reimbursement code may be provided by a healthcareprovider, a healthcare administrator, and/or other users. Or, asdiscussed herein, the processed reimbursement code may be provided via amachine-learning method for analyzing historical surgical procedures andidentifying processed reimbursement codes that were used for historicalsurgical procedures. In various embodiments, a processed reimbursementcode may differ from at least one of the outputted reimbursement codes.This may occur after manual identification of a correct code by ahealthcare professional, or after further machine learning analysisdetermines a more accurate reimbursement code candidate.

As previously described, some embodiments may include using a machinelearning model to detect, in the historical video footage, the at leastone plurality of medical instruments, plurality of anatomicalstructures, or plurality of interactions between medical instruments andanatomical structures. As described herein, the machine-learning methodmay be any suitable image recognition method trained to recognize one ormore medical instruments, anatomical structures, and interactionsbetween the instruments and the structures. In an example embodiment, amachine-learning method may employ multiple image recognitionalgorithms, with each algorithm trained to recognize a particularmedical instrument or a particular anatomical structure.

Aspects of disclosed embodiments may further include analyzing videoframes captured during a surgical procedure to determine a condition ofan anatomical structure of a patient and determining at least onereimbursement code associated with the surgical procedure based on thedetermined condition of the anatomical structure. Procedures performedon anatomical structures in poor condition, for example, may justifyhigher reimbursement than procedures performed on anatomical structuresin better condition. In an example embodiment, a machine-learning methodmay be used based on information obtained from various sensors fordetermining the condition of an anatomical structure of a patient. Acondition of an anatomical structure may be determined based on observedvisual characteristics of the anatomical structure such as a size,color, shape, translucency, reflectivity of a surface, fluorescence,and/or other image features. A condition may be based on one or more ofthe anatomical structure, temporal characteristics (motion, shapechange, etc.) for the anatomical structure, sound characteristics (e.g.,transmission of sound through the anatomical structure, sound generatedby the anatomical structure, and/or other aspects of sound), imaging ofthe anatomical structure (e.g., imaging using x-rays, using magneticresonance, and/or other means), or electromagnetic measurements of thestructure (e.g., electrical conductivity of the anatomical structure,and/or other properties of the structure). Image recognition can be usedto determine anatomical structure condition. Additionally oralternatively, other specialized sensors (e.g., magnetic field sensors,electrical resistance sensors, sound sensors or other detectors) may beused in condition determination.

In various embodiments, upon determining a condition of an anatomicalstructure, a reimbursement code may be identified using, for example, asuitable machine-learning mode. For instance, the machine-learning modelmay take a condition of an anatomical structure as one possibleparameter for determining one or more reimbursement codes. FIG. 21 showsan example system 2101 for determining one or more reimbursement codes(e.g., codes 2137, as schematically shown in FIG. 21 ). In an exampleembodiment, surgical footage 2111 may be processed by a machine-learningmethod 213, and method 213 may identify medical instruments 2116,anatomical structures 2118, interactions of medical instrument andanatomical structures 2120, and various parameters 2122 (herein alsoreferred to as properties or characteristics) such as parameters C1-CNdescribing instruments 2116, anatomical structures 2118, interactions2120, and any other information that might impact a reimbursement code.An example parameter C1 may be a size of the incision, parameter C2 maybe a condition of an anatomical structure (e.g., a size, a color, ashape, and/or other image property of the anatomical structure), andparameter CN may be a location at which an example medical instrumentinteracted with an example anatomical structure. Information aboutmedical instruments 2116, anatomical structures 2118, interactions 2120,and parameters 2122 may be used as an input 2110 for a computer-basedsoftware application, such as a machine-learning model 2135. Model 2135may process input 2110 and output one or more reimbursement codesassociated with a segment of a surgical procedure having information asdescribed by input 2110.

In some of the embodiments, analyzing surgical images to determineinsurance reimbursement may include analyzing video frames capturedduring a surgical procedure to determine a change in a condition of ananatomical structure of a patient during the surgical procedure, anddetermining the at least one reimbursement code associated with thesurgical procedure based on the determined change in the condition ofthe anatomical structure. A process of analyzing video frames todetermine a change in the condition of an anatomical structure of thepatient may be performed using any suitable machine-learning method. Forexample, the change in a condition of an anatomical structure mayinclude a change in shape, color, size, location, and/or other imageproperty of the anatomical structure. Such change may be determined byimage recognition algorithm as described herein and consistent withvarious described embodiments. An image recognition algorithm mayidentify an anatomical structure in a first set of frames of surgicalprocedure, identify an anatomical structure in a second set of frames ofsurgical procedure and evaluate if the anatomical structure changed fromthe first to the second set of frames. If the change is observed, theimage recognition algorithm may qualify the change by assigning a changerelated identifier. By way of a few examples, the change-relatedidentifier may be a string “removed tumor,” “removed appendix,” “carotidarteries with a removed blockage,” and/or other data describing achange. Change-related identifiers may be selected from a list ofpreconfigured identifiers, and may include one of the parameters of asurgical procedure, such as parameters C1-CN, as shown in FIG. 21 , usedas part of an input for a machine-learning model (e.g., model 2135) tooutput reimbursement codes (e.g., codes 2137). In this way, areimbursement code may be associated with the surgical procedure basedon the determined change in the condition of the anatomical structure.

Disclosed embodiments may also include analyzing the video framescaptured during a surgical procedure to determine usage of a particularmedical device, and determining at least one reimbursement codeassociated with the surgical procedure based on the determined usage ofthe particular medical device. The use of certain medical instrumentsmay impact reimbursement codes. For example, the detection of certaindisposable medical devices may trigger reimbursement for those devices.Or the use of a costly imaging machine (MRI, CT, etc.), may triggerreimbursement for usage of that device. Moreover, the usage of certaindevices, regardless of their cost, can be correlated to the complexity,and therefore the cost of a procedure.

Some embodiments may further include analyzing video frames capturedduring a surgical procedure to determine a type of usage of a particularmedical device, and in response to a first determined type of usage,determining at least a first reimbursement code associated with thesurgical procedure; and in response to a second determined type ofusage, determining at least a second reimbursement code associated withthe surgical procedure, the at least a first reimbursement codediffering from the at least a second reimbursement code. A type of usagemay be any technique or manipulation of the medical device, such asincision making, imaging, suturing, surface treatment, radiationtreatment, chemical treatment, cutting, and/or other treatmentmodalities. In various embodiments, the type of usage may be analyzed byanalyzing video frames captured during a surgical procedure (i.e.,surgical footage).

Consistent with various embodiments described herein, detection of typeof usage may occur through image recognition, as previously discussed.In some cases, the location of a device relative to an anatomicalstructure may be used to determine the interaction of the medical devicewith the anatomical structure. In various embodiments, for each type oftreatment using a medical device, a corresponding reimbursement code maybe used. In some cases, the same medical device may be used fordifferent types of treatments that may have different associatedreimbursement codes. For example, forceps can be used first to clamp ananatomical structure, and then used to extract an anatomical structure.In some examples, a type of usage of a particular medical device may bedetermined by analyzing video frames captured during a surgicalprocedure. For example, a machine learning model may be trained usingtraining example to determine types of usages of medical devices fromimages and/or videos of surgical procedures, and the trained machinelearning model may be used to analyze the video frames captured during asurgical procedure and determine the type of usage of the particularmedical device. An example of such training example may include an imageand/or a video of at least a portion of a surgical procedure, togetherwith a label indicating the type of usage of a particular medical devicein the surgical procedure.

In some examples, a machine learning model may be trained using trainingexamples to determine reimbursement codes for surgical procedures basedon information related to the surgical procedures. An example of suchtraining example may include information related to a particularsurgical procedure, together with a label indicating the desiredreimbursement code for the particular surgical procedure. Somenon-limiting examples of such information related to the surgicalprocedures may include images and/or videos of the surgical procedure,information based on an analysis of the images and/or videos of thesurgical procedure (some non-limiting examples of such analysis andinformation are described herein), an anatomical structure related tothe surgical procedure, a condition of an anatomical structure relatedto the surgical procedure, a medical instrument used in the surgicalprocedure, an interaction between a medical instrument and an anatomicalstructure in the surgical procedure, phases of the surgical procedure,events that occurred in the surgical procedure, information based on ananalysis of a post-operative report of the surgical procedure, and soforth. Further, in some examples, the trained machine learning model maybe used to analyze the video frames captured during the surgicalprocedure to determine the at least one reimbursement code associatedwith the surgical procedure. In other examples, the trained machinelearning model may be used to determine the at least one reimbursementcode associated with the surgical procedure based on any informationrelated to the surgical procedure, such as at least one interactionbetween at least one medical instrument and at least one anatomicalstructure in the surgical procedure (for example, the at least oneinteraction between the at least one medical instrument and the at leastone anatomical structure identified by analyzing the video framescaptured during the surgical procedure), an analysis of a postoperativesurgical report of the surgical procedure, a condition of an anatomicalstructure of the patient (for example, a condition of an anatomicalstructure of the patient determined by analyzing the video framescaptured during the surgical procedure), a change in a condition of ananatomical structure of the patient during the surgical procedure (forexample, a change in a condition of an anatomical structure of thepatient during the surgical procedure determined by analyzing the videoframes captured during the surgical procedure), a usage of a particularmedical device (for example, a usage of a particular medical devicedetermined by analyzing the video frames captured during the surgicalprocedure), a type of usage of a particular medical device (for example,a type of usage of the particular medical device determined by analyzingthe video frames captured during the surgical procedure), an amount of amedical supply of a particular type used in the surgical procedure (forexample, an amount of a medical supply of the particular type used inthe surgical procedure and determined by analyzing the video framescaptured during the surgical procedure), and so forth.

Additionally, embodiments may include analyzing video frames capturedduring a surgical procedure to determine an amount of a medical supplyof a particular type used in the surgical procedure and determining theat least one reimbursement code associated with the surgical procedurebased on the determined amount. In an example embodiment, the amount ofa medical supply of a particular type may be determined using an imagerecognition algorithm for observing video frames of a surgical procedurethat may indicate an amount of a medical supply that was used during thesurgical procedure. The medical supply may be any material used duringthe procedure, such as medications, needles, catheters, or any otherdisposable or consumable material. The amount of supply may bedetermined from video frames of a surgical procedure. For example, theamount of medication used by a patient may be determined by observing anintravenous (IV) apparatus for supplying medications and fluids to apatient. Bags of intravenous blood or fluids may be counted as they arereplaced. In various embodiments, a suitable machine-learning model maybe used to identify an amount of a medical supply of a particular typeused during, prior, and/or after the surgical procedure, and determiningat least one reimbursement code associated with the surgical procedurebased on the determined amount. The machine-learning model may betrained using historical surgical footage of a historical surgicalprocedure and historical data for amounts of a medical supply usedduring the historical surgical procedure. In some examples, an amount ofa medical supply of a particular type used in a surgical procedure maybe determined by analyzing video frames captured during the surgicalprocedure. For example, a machine learning model may be trained usingtraining example to determine amounts of medical supplies of particulartypes used in surgical procedures from images and/or videos of surgicalprocedures, and the trained machine learning model may be used toanalyze the video frames captured during a surgical procedure anddetermine the amount of the medical supply of the particular type usedin the surgical procedure. An example of such training example mayinclude an image and/or a video of at least a portion of a particularsurgical procedure, together with a label indicating the amount of themedical supply of the particular type used in the particular surgicalprocedure.

Aspects of a method of analyzing surgical images to determine insurancereimbursement code are illustrated by an example process 2201, as shownin FIG. 22 . At step 2211 of process 2201, a method may includeaccessing video frames captured during a surgical procedure on apatient. Video frames may be captured using any suitable image sensorsand may be accessed using a machine-learning method and/or a healthcareprovider, as discussed above. At step 2213, the method may includeanalyzing the video frames captured during the surgical procedure toidentify in the video frames at least one medical instrument, at leastone anatomical structure, and at least one interaction between the atleast one medical instrument and the at least one anatomical structure,as described above. For example, the frames may be analyzed using asuitable machine-learning method, such as an image recognitionalgorithm, as previously discussed. At step 2215, the method may includeaccessing a database of reimbursement codes correlated to medicalinstruments, anatomical structures, and interactions between medicalinstruments and anatomical structures. At step 2217, the method mayinclude comparing the identified at least one interaction between the atleast one medical instrument and the at least one anatomical structurewith information in the database of reimbursement codes to determine atleast one reimbursement code associated with the surgical procedure, aspreviously described, and at step 2219, the method may includeoutputting the at least one reimbursement code for use in obtaining aninsurance reimbursement for the surgical procedure.

As previously discussed, the present disclosure relates to methods andsystems for analyzing surgical images to determine insurancereimbursement, as well as a non-transitory computer-readable media thatmay include instructions that, when executed by at least one processor,cause the at least one processor to execute operations enablinganalyzing surgical images to determine insurance reimbursement, asdescribed above.

Disclosed systems and methods may involve analyzing surgical footage toidentify features of surgery, patient conditions, and surgicalintraoperative events to obtain information for populating thepostoperative report. A postoperative report may be populated byanalyzing surgical data obtained from a surgical procedure to identifyfeatures of surgery, patient conditions, and surgical intraoperativeevent and extracting information from the analyzed data for populatingthe postoperative report. Therefore, there is a need for analyzingsurgical data, and extracting information from the surgical data thatmay be used for populating a postoperative report.

Aspects of this disclosure may relate to populating a post-operativereport of a surgical procedure, including methods, systems, devices, andcomputer readable media. For ease of discussion, a method is describedbelow, with the understanding that aspects of the method apply equallyto systems, devices, and computer readable media. For example, someaspects of such a method may occur electronically over a network that iseither wired, wireless, or both. Other aspects of such a method mayoccur using non-electronic means. In the broadest sense, the method isnot limited to particular physical and/or electronic instrumentalities,but rather may be accomplished using many differing instrumentalities.

Consistent with disclosed embodiments, a method for populating apost-operative report of a surgical procedure may include receiving aninput of an identifier of a patient. Further, the method may includereceiving an input of an identifier of a health care provider. Apost-operating report may be any suitable computer-based or paper-basedreport documenting a surgical procedure. In various embodiments, apost-operative report may include multiple frames of surgical footage,audio data, image data, text data (e.g., doctor notes) and the like. Inan example embodiment, a post-operative report may be populated,partially populated, or not populated. For example, the post-operativereport may contain fields (e.g., regions of the report) for holdingvarious details obtained during the surgical procedure. In an exampleembodiment, at least some fields may have an associated characteristic(also referred to as a field name) that may determine what type ofinformation can be entered in the field. For instance, a field with anassociated name “Name of a Patient” may allow a name of a patient to beentered in that field. A field named “Pulse Plot” may be a field fordisplaying a pulse of a patient during the surgical procedure plotted asa function of time. In various embodiments, when the report is notpopulated, all the fields in the report may be empty; when the report ispartially populated, some of the fields may contain information obtainedfrom a surgical procedure; and when the report is fully populated (ormostly populated) the vast majority of the fields may containinformation relating to an associated surgical procedure. In someexamples, at least part of a post-operative report may have a free formformat, allowing users and/or automatic processes to enter data invarious organizations and/or formats, such as free text, which in someexamples may include other elements embedded freely in the free text oraccompanying it, such as links to external elements, images, videos,audio recordings, digital files, and so forth. It is appreciated thatany detail described herein as included in a post-operative report in aparticular field may be equally included in a post-operative report aspart of such free textual information, embedded in the free text, oraccompanying it.

An example post-operative report 2301 is shown in FIG. 23 . Report 2301may contain multiple fields, sections, and subsections. Different fieldsmay contain different types of information. For example, field 2310 maycontain a name of the surgical procedure, field 2312 may contain a nameof a patient and field 2314 may contain a name of a healthcare provider.Field 2316 may include a name of a phase of a surgical procedure, andfield 2318 may include a sequential number of a phase (e.g., a firstphase of a surgical procedure). Multiple instances of fields 2314 and/or2316 may be included in post/operative report 2301, to described aplurality of phases of the surgical procedure. Report 2301 may include asection 2315 that may describe a particular event during a surgicalprocedure. Multiple sections for describing multiple events may bepresent in report 2301. One or more of the events may be connected to aparticular surgical phase, while other events may not be connected toany surgical phase. In an example embodiment, section 2315 may include afield 2320 containing a name of the event, field 2321A containing astarting time for the event, field 2321B containing a finishing time forthe event, and field 2324 containing description of the event (e.g.,field 2324 may contain notes from a healthcare provider describing theevent). Section 2315 may include subsection 2326 for containing fieldsfor images such as fields IMAGE 1 through IMAGE N, as well as subsection2328 for containing event-related surgical footage. For example,subsection 2328 may include fields V1-VN. Additionally, section 2315 mayinclude subsection 2329 that may contain links to various other datarelated to a surgical procedure. In various embodiments, apost-operative report may be partitioned into different portionsindicated by tabs 2331 and 2333, as shown in FIG. 23 . For example, whena user selects tab 2331, information related to a first portion of asurgical report may be displayed, and when a user selects tab 2333,information related to a second portion of a surgical report may bedisplayed. In various embodiments, a surgical report may include anysuitable number of portions.

FIG. 23 also shows that information may be uploaded into report 2301,via an upload input form 2337. For example, a user may click on a field(e.g., field V1, as shown in FIG. 23 ), and form 2337 may be presentedto the user for uploading data for the field V1. In various embodiments,fields, sections, subsections, and tabs, as shown in FIG. 23 are onlyillustrative, and any other suitable fields, sections, subsections, andtabs may be used. Furthermore, a number and types of fields, sections,subsections, and tabs may depend on information entered inpost-operative report 2301.

In various embodiments, information for populating at least part of apost-operative report may be obtained from surgical footage of asurgical procedure. Such information may be referred to as image-basedinformation. Additionally, information about a surgical procedure may beobtained from notes of a healthcare provider or a user, previously filedforms for a patient (e.g., a medical history for the patient), medicaldevices used during a surgical procedure, and the like. Such informationmay be referred to as auxiliary information. In an example embodiment,auxiliary information may include vital signs, such as pulse, bloodpressure, temperature, respiratory rate, oxygen levels, and the likereported by various medical devices used during a surgical procedure.Image-based information and auxiliary information may be processed by asuitable computer-based software application and the processedinformation may be used to populate a post-operative report. Forexample, FIG. 24A shows an example of a process 2401 for processinginformation and populating a post-operative report 2301. In an exampleembodiment, image-based information 2411 and auxiliary information 2413may be used as an input to a computer-based software application 2415,and application 2415 may be configured to process information 2411 and2413, extract data for various fields present in a post-operative report(e.g., report 2301, as shown in FIG. 24A), and populate the variousfields (as schematically indicated by arrows 2430A-2430D). FIG. 24Bshows an example system 2402 for processing information and populating apost-operative report 2301, 2402 may differ from system 2401 in thatvarious data processed by application 2415 may be stored in a database2440 prior to populating post-operative report 2301. By storing data indatabase 2440, the data may be easily accessed for use in generatingvarious other reports. Database 2440 may be configured to execute asoftware application for mapping data from database 2440 to fields ofreport 2301 as schematically shown by arrows 2431A-2431D.

As described above, embodiments for populating a post-operative reportmay include receiving an input of an identifier of a patient and ahealthcare provider. The identifier of a patient may be any suitabledata or physical indicator (e.g., a patient's name, date of birth,social security number or other government identifier, patient number orother unique code, patient image, DNA sequence, a vocal ID, or any otherindicator that uniquely identifies the patient. In some cases, a groupof identifiers may be used as a combined identifier. In an exampleembodiment, an identifier may be an alphanumerical string that uniquelyidentifies the patient.

In various embodiments, of the patient identifier may be received as aninput. This may occur using any suitable process of transmission (e.g.,a process of transmission of data over a wired or wireless network, aprocess of transmission of data using a suitable input device such as akeyboard, mouse, joystick, and the like). In some cases, “receiving aninput” may include receipt through mail or courier (e.g., a paperdocument delivered in person).

Similar to the patient identifier, the identifier of a health careprovider may be any suitable indication of identity, such as a name, acode, an affiliation, an address, an employee number, a PhysicianLicense Number, or any other mechanism of identifying the healthcareprovider. In an example embodiment, an identifier may be analphanumerical string that uniquely identifies the healthcare provider.

Disclosed embodiments may further include receiving an input of surgicalfootage of a surgical procedure performed on the patient by the healthcare provider. Surgical footage may be received as input by acomputer-based software application for analyzing the input (e.g.,application 2415, as shown in FIG. 24A) and/or, in some cases, receivingan input may include receiving the input by a healthcare professional ora user. This may occur, for example, when a healthcare professional orthe user uploads the video footage from a storage location and/ordirectly from sensors capturing the video footage.

The surgical footage of a surgical procedure may include any form ofrecorded visual data, including recorded images and/or video data, whichmay also include sound data. Visual data may include a sequence of oneor more images captured by image sensors, such as cameras 115, 121, 123,and/or 125, as described above in connection with FIG. 1 . Some of thecameras (e.g., cameras 115, 121, and 125) may capture video/image dataof operating table 141, and camera 121 may capture video/image data of asurgeon 131 performing the surgery. In some cases, cameras may capturevideo/image data associated with surgical team personnel, such as ananesthesiologist, nurses, surgical tech and the like located inoperating room 101.

In various embodiments, image sensors may be configured to capture thesurgical footage by converting visible light, x-ray light (e.g., viafluoroscopy), infrared light, or ultraviolet light to images, a sequenceof images, videos, and the like. The image/video data may be stored ascomputer files using any suitable format such as JPEG, PNG, TIFF, AudioVideo Interleave (AVI), Flash Video Format (FLV), QuickTime File Format(MOV), MPEG (MPG, MP4, M4P, etc.), Windows Media Video (WMV), MaterialExchange Format (MXF), and the like.

A surgical procedure may include any medical procedure associated withor involving manual or operative procedures on a patient's body.Surgical procedures may include cutting, abrading, suturing, or othertechniques that involve physically changing body tissues and/or organs.Surgical procedures may also include diagnosing patients oradministering drugs to patients. Some examples of such surgicalprocedures may include a laparoscopic surgery, a thoracoscopicprocedure, a bronchoscopic procedure, a microscopic procedure, an opensurgery, a robotic surgery, an appendectomy, a carotid endarterectomy, acarpal tunnel release, a cataract surgery, a cesarean section, acholecystectomy, a colectomy (such as a partial colectomy, a totalcolectomy, etc.), a coronary angioplasty, a coronary artery bypass, adebridement (for example of a wound, a burn, an infection, etc.), a freeskin graft, a hemorrhoidectomy, a hip replacement, a hysterectomy, ahysteroscopy, an inguinal hernia repair, a knee arthroscopy, a kneereplacement, a mastectomy (such as a partial mastectomy, a totalmastectomy, a modified radical mastectomy, etc.), a prostate resection,a prostate removal, a shoulder arthroscopy, a spine surgery (such as aspinal fusion, a laminectomy, a foraminotomy, a discectomy, a diskreplacement, an interlaminar implant, etc.), a tonsillectomy, a cochlearimplant procedure, brain tumor (for example meningioma, etc.) resection,interventional procedures such as percutaneous transluminal coronaryangioplasty, transcatheter aortic valve replacement, minimally Invasivesurgery for intracerebral hemorrhage evacuation, or any other medicalprocedure involving some form of incision. While the present disclosureis described in reference to surgical procedures, it is to be understoodthat it may also apply to other forms of medical procedures orprocedures generally.

In various embodiments, the surgical procedure may be performed on thepatient by a healthcare provider, with the patient being identified bythe identifier, as described above. The healthcare provider may be aperson, a group of people, an organization, or any entity authorized toprovide health services to a patient. For example, the healthcareprovider may be a surgeon, an anesthesiologist, a nurse practitioner, ageneral pediatrician, or any other person or a group of people that maybe authorized and/or able to perform a surgical procedure. In variousembodiments, the healthcare provider may be a surgical team forperforming the surgical procedure and may include a head surgeon, anassistant surgeon, an anesthesiologist, a nurse, a technician, and thelike. The healthcare provider may administer a surgical procedure,assist with the surgical procedure for a patient and the like. Ahospital, clinic, or other organization or facility may also becharacterized as a healthcare provider, consistent with disclosedembodiments. Likewise, a patient may be a person (or any livingcreature) on whom a surgical procedure is performed.

Aspects of disclosed embodiments may include analyzing a plurality offrames of the surgical footage to derive image-based information forpopulating a post-operative report of the surgical procedure. In variousembodiments, image-based information may include information aboutevents that occurred during the surgical procedure, information aboutphases of the surgical procedure, information about surgical tools usedduring the surgical procedure, information about anatomical structureson which the surgical procedure was performed, data from various devices(e.g., vital signs, such as pulse, blood pressure, temperature,respiratory rate, oxygen levels, and the like), or any other suitableinformation that may be obtained from the images and may be applicableto be documented in the post-operative report. Some other non-limitingexamples of information based on an analysis of surgical footage and/oralgorithms for analyzing the surgical footage and determining theinformation are described in this disclosure.

In various embodiments, the image-based information may be derived fromthe surgical footage using any suitable trained machine-learning model(or other image recognition algorithms) for identifying events, phasesof surgical procedures, surgical tools, anatomical structures within thesurgical footage, and the like, for example as described above. In somecases, the machine learning method may identify various properties ofevents, phases, surgical tools, anatomical structures, and the like. Forexample, a property of an event such as an incision may include thelength of the incision, and a property of an anatomical structure mayinclude a size of the structure or shape of the structure. In variousembodiments, any suitable properties may be identified using amachine-learning method, for example as described above, and onceidentified may be used to populate a surgical report.

In various embodiments, the derived image-based information may be usedfor populating a post-operative report of the surgical procedure. Aprocess of populating the post-operative report may include populatingfields of the report with information specific to the fields. In anexample embodiment, populating a post-operative report may be done by acomputer-based application (e.g., application 2415, as shown in 24A).For example, the computer-based application may be configured toretrieve a field from the post-operative report, determine a nameassociated with the field, determine what type of information (e.g.,image-based information, or any other suitable information) needs to beentered in the field based on a determined name, and retrieve suchinformation from either surgical footage or from auxiliary information(e.g., auxiliary information 2413, as shown in FIG. 24A). In an exampleembodiment, retrieving information may include deriving image-basedinformation from the surgical footage. For example, if the field name“Surgical Tools Used,” retrieving information may include using an imagerecognition algorithm for identifying (in the surgical footage) surgicaltools used during the surgical procedure, and populating the surgicalreport with the names of the identified tools. Thus, derived image-basedinformation may be used to populate the post-operative report of thesurgical procedure. Other examples of image-based information that maybe used to populate the report may include the starting and ending timesof a procedure or portion thereof, complications encountered, conditionsof organs, and other information that may be derived through analysis ofvideo data. These might also include, characteristics of a patient,characteristics of one or more healthcare providers, information aboutan operating room (e.g., the type of devices present in the operatingroom, type of image sensors available in the operating room, etc.), orany other relevant data.

Aspects of a method of populating a post-operative report of a surgicalprocedure are illustrated by an example process 2501, as shown in FIG.25 . At step 2511 of process 2501, the method may include receiving aninput of an identifier of a patient, and at step 2513, the method mayinclude receiving an input of an identifier of a health care provider,as described above. At step 2515, the method may include receiving aninput of surgical footage of a surgical procedure performed on a patientby a health care provider. Receiving the input of surgical footage mayinclude receiving the input by a suitable computer-based softwareapplication or a healthcare professional, as discussed above. At step2517, the method may include analyzing a plurality of frames of thesurgical footage to derive image-based information for populating apost-operative report of the surgical procedure, as described herein,and at step 2519, the method may include causing the derived image-basedinformation to populate the post-operative report of the surgicalprocedure, as previously described.

Aspects of a method of populating a post-operative report of a surgicalprocedure may include analyzing the surgical footage to identify one ormore phases of the surgical procedure. The phases may be distinguishedfrom each other automatically based on a training model trained todistinguish one portion of a surgical procedure from another, forexample as described herein.

For the purposes of the present disclosure, a phase may refer to aparticular period or stage of a process or series of events.Accordingly, a surgical phase may refer to a sub-portion of a surgicalprocedure. For example, surgical phases of a laparoscopiccholecystectomy surgery may include trocar placement, preparation,calot's triangle dissection, clipping and cutting of cystic duct andartery, gallbladder dissection, gallbladder packaging, cleaning andcoagulation of liver bed, gallbladder retraction, and so forth. Inanother example, surgical phases of a cataract surgery may includepreparation, povidone-iodine injection, corneal incision, capsulorhexis,phaco-emulsification, cortical aspiration, intraocularlens implantation,intraocular-lens adjustment, wound sealing, and so forth. In yet anotherexample, surgical phases of a pituitary surgery may include preparation,nasal incision, nose retractor installation, access to the tumor, tumorremoval, column of nose replacement, suturing, nose compressinstallation, and so forth. Some other examples of surgical phases mayinclude preparation, incision, laparoscope positioning, suturing, and soforth.

In some examples, the user may identify a phase by marking a section ofthe surgical footage with a word/sentence/string that identifies a nameor a type of a phase. The user may also identify an event, procedure, ordevice used, which input may be associated with particular video footage(e.g., for example through a lookup table or other data structure). Theuser input may be received through a user interface of a user device,such as a desktop computer, a laptop, a tablet, a mobile phone, awearable device, an internet of things (IoT) device, or any other meansfor receiving input from a user. The interface may provide, for example,one or more drop-down menus with one or more pick lists of phase names;a data entry field that permits the user to enter the phase name and/orthat suggests phase names once a few letters are entered; a pick listfrom which phase names may be chosen; a group of selectable icons eachassociated with a differing phase, or any other mechanism that allowsusers to identify or select a phase.

In some embodiments, analyzing the surgical procedure to identify one ormore phases of the surgical procedure may involve using computeranalysis (e.g., a machine-learning model) to analyze frames of the videofootage, for example as described above. Computer analysis may includeany form of electronic analysis using a computing device. In someembodiments, computer analysis may include using one or more imagerecognition algorithms to identify features of one or more frames of thevideo footage. Computer analysis may be performed on individual framesor may be performed across multiple frames, for example, to detectmotion or other changes between frames.

In some embodiments, analyzing the surgical procedure to identify atleast one phase of the surgical procedure may involve associating a namewith at least one phase. For example, if the identified phase includesgallbladder dissection, a name “gallbladder dissection” may beassociated with that phase. In various embodiments, derived image-basedinformation (derived from surgical footage of a surgical procedure byidentifying a phase), may include an associated phase name, as describedabove.

Further, aspects of a method of populating a post-operative report of asurgical procedure may include identifying a property of at least onephase of identified phases. A property of a phase may be anycharacteristics of a phase such as a duration of the phase, a place ofthe phase in a sequence of phases during the surgical procedure, a phasecomplexity, an identification of a technique used, information relatedto medical instruments used in the phase, information related to actionsperformed in the phase, changes in a condition of an anatomicalstructure during the phase, or any other information that maycharacterize the phase. A phase property may be expressed in the form ofan alphanumerical string. For instance, “a first phase” may identify thephase as a first phase in a sequence of phases during a surgicalprocedure, “one hour” may describe that the phase has a duration of onehour, “bronchoscopy” may identify a phase as a bronchoscopy, and thelike. Additionally or alternatively, a property of a phase may benon-textural data (e.g., image, audio, numerical, and/or video data)collected during a surgical procedure. For example, a representativeimage of an anatomical structure (or surgical instrument, or aninteraction of a surgical instrument with an example anatomicalstructure) performed during a phase of a surgical procedure may be usedas a property of a phase. In one example, a machine learning model maybe trained using training examples to identify properties of surgicalphases from images and/or videos. An example of such training examplemay include an image and/or a video of at least a portion of a surgicalphase of a surgical procedure, together with a label indicating one ormore properties of the surgical phase. Some non-limiting examples ofsuch properties may include a name of the surgical phase, a textualdescription of the surgical phase, or any other property of a surgicalphase described above. Further, in some examples, the trained machinelearning model may be used to analyze the surgical footage to identifythe property of the at least one phase of identified phases. In variousembodiments, the derived image-based information (used for populatingthe surgical record) may be based on the identified at least one phaseand the identified property of the at least one phase. For example, thecombination of both the phase and the property together may enable thephase to be recorded in a way that is more meaningful. For example,during a phase of suturing of a valve, if an intraoperative leak isdetected (a property of the phase), the phase/property combination maybe recorded in the surgical record. In some cases, the derivedimage-based information may include a segment of a video captured duringthe phase of the surgical procedure.

Aspects of a method of populating a post-operative report of a surgicalprocedure may include determining at least a beginning of the at leastone phase; and wherein the derived image-based information is based onthe determined beginning. The beginning of at least one phase may bedetermined by performing a computer image analysis on surgical footage,for example as described above. For example, using a trained machinelearning model (such as a recurrent convolutional neural network), thebeginning of a particular phase may be distinguished from the end of aprior phase, and the location may be identified and stored in thesurgical record. In another example, a phase may start when a particularmedical instrument first appears in the video footage, and an objectdetection algorithm may be used to identify the first appearance of theparticular medical instrument in the surgical footage.

In some cases, a time marker may be associated with the at least onephase, and the derived image-based information may include the timemarker associated with the at least one phase. The time marker may berecorded in a number of ways, including, a time elapsed from thebeginning of the surgical procedure, the time as measured by the time ofday, or a time as it relates to some other intraoperative recorded time.In various embodiments, a time marker may be associated with thebeginning of each identified phase (e.g., a time marker may beassociated with the beginning location of the surgical phase within thesurgical footage). The time marker may be any suitable alphanumericalidentifier, or any other data identifier (e.g., an audio signal or animage) and may include information about a time (and/or possibly a timerange), associated with the beginning of the identified phase.

An example surgical event, such as an incision, may be detected usingaction detection algorithms, for example as discussed above. Such anidentified surgical event may identify a beginning of a surgical phase.In an example embodiment, an event that begins a surgical phase may bedetected based on machine learning techniques. For example, a machinelearning model may be trained using historical surgical footageincluding known events that begin the surgical phase.

Further, disclosed embodiments may include determining at least anending of the at least one phase, and derived image-based informationmay be based on the determined ending. The end of the surgical phase maybe determined by detecting an end location of the surgical phase withinthe surgical footage. In various embodiments, a time marker may beassociated with the end of each identified phase (e.g., the time markermay be associated with the end location of the surgical phase within thesurgical footage). As discussed above, the ending marker may be recordedin the same manner as the starting marker, and may be characterized byany suitable alphanumerical identifier, or any other data identifier.For example, the surgical footage may be analyzed to identify thebeginning of a successive surgical phase, and the ending of one phasemay be identical to the beginning of the successive surgical phase. Inanother example, a phase may end when a particular medical instrumentlast appears in the video footage, and an object detection algorithm maybe used to identify the last appearance of the particular medicalinstrument in the surgical footage.

Embodiments for automatically populating a post-operative report of asurgical procedure may also include transmitting data to a health careprovider, the transmitted data, including a patient identifier andderived image-based information. During or after a surgical procedure,video captured during the surgical procedure may be transmitted to ahealthcare provider for populating the patient's associated surgicalrecord. In order to ensure that the video populates the appropriaterecord, the patient identifier may accompany the video in thetransmission. In some embodiments, this may enable the surgical recordto be automatically updated with the video, without human intervention.In other embodiments, either on the transmission and/or the receivingend, a human may select the video for transmission, or accept the videofor incorporation into the patient's medical record. In some cases,transmitting data may involve mailing (or delivering in person) aphysical copy (e.g., a paper copy, a CD-ROM, a hard drive, a DVD, a USBdrive, and the like) of documents describing the data. Additionally oralternatively, transmitting data may include transmitting data to atleast one of a health insurance provider or a medical malpracticecarrier.

Aspects of the disclosure may include analyzing the surgical footage toidentify at least one recommendation for post-operative treatment; andproviding the identified at least one recommendation. As describedearlier, surgical footage may be analyzed in various ways (e.g., using amachine-learning method, by a healthcare provider, and the like). Invarious embodiments, a machine-learning method may be configured notonly to recognize events within the video frames but also configured toform conclusions about various aspects of the surgical procedure basedon an analysis of surgical footage. For example, post-operative woundcare may vary depending on the nature of the surgical wound. Videoanalysis might determine that nature, and might also provide arecommendation for post-operative treatment of the wound site. Suchinformation may be transmitted to and stored in the surgical record. Insome cases, the machine-learning method may identify intraoperativeevents (e.g., adverse events) and may provide indications for theseevents for which specific post-operative treatments are needed. This maybe analyzed through machine learning and the recommendation forpost-operative treatment may be automatically provided. In one example,in response to a first surgical event identified in the surgicalfootage, a first recommendation for post-operative treatment may beidentified, and in response to a second event identified in the surgicalfootage, a second recommendation for post-operative treatment may beidentified, the second recommendation may differ from the firstrecommendation. In one example, in response to a first condition of ananatomical structure identified in the surgical footage, a firstrecommendation for post-operative treatment may be identified, and inresponse to a second condition of the anatomical structure identified inthe surgical footage, a second recommendation for post-operativetreatment may be identified, the second recommendation may differ fromthe first recommendation. In some examples, a machine learning model maybe trained using training examples to generate recommendations forpost-operative treatment from surgical images and/or surgical videos,and the trained machine learning model may be used to analyze thesurgical footage and identifying the at least one recommendation forpost-operative treatment. An example of such training example mayinclude an image or a video of at least a portion of a surgicalprocedure, together with a label indicating the desired recommendationsfor post-operative treatment corresponding to the surgical procedure.

Such recommendations may include suggesting physical therapy,medications further physical examination, a follow on surgicalprocedure, and the like. In some cases, recommendations may not directlyrelate to medical activities but may include diet recommendations, sleeprecommendations, recommendations for physical activity, orrecommendations for stress management. In various embodiments, theidentified recommendation may be provided to a healthcare professionalresponsible for a post-operative treatment for the patient. Additionallyor alternatively, the recommendation may be provided to a third partywhich may be a patient, a family member, a friend, and the like.

In one embodiment, an analysis of surgical footage may includeidentifying that during a given time of a surgical procedure, a surgeonmay have worked too closely to intestines of a patient, for example,using an energy device. When such an event is identified (for exampleusing an object detection algorithm, using a trained machine learningmodel, etc.), a notification (e.g., a push notification) may be send toalert a surgeon (or any other healthcare professional supervising apost-operative treatment of a patient) to further analyze the surgicalfootage and to have special procedures planned to avoid a catastrophicpost-operative event (e.g., bleeding, cardiac arrest, and the like).

In various embodiments, populating a post-operative report of a surgicalprocedure may include enabling a health care provider to alter at leastpart of derived image-based information in the post-operative report.For example, the healthcare provider (also referred to as a healthcareprofessional) may access a post-operative report via a softwareapplication configured to display information in the post-operativereport. In various embodiments, a healthcare professional may be enabledto alter some or all fields within the post-operative report. In someembodiments, particular fields may be locked as unalterable withoutadministrative rights. Examples of alterable fields may be thosecontaining text-based data (e.g., alterable by inputting new data viakeyboard, mouse, microphone, and the like), image data (e.g., byuploading one or more images related to a surgical procedure, overlayinginformation over the one or more images, etc.), video data (e.g., byuploading one or more videos related to a surgical procedure overlayinginformation over one or more frames of the one or more videos, etc.),audio data (e.g., the audio data captured during a surgical procedure),and the like.

In various embodiments, updates to a post-operative report may betracked using a version tracking system. In an example embodiment, theversion tracking system may maintain all data that was previously usedto populate a post-operative report. The version tracking system for maybe configured to track differences between different versions of apost-operative report, and may be configured to track information abouta party (e.g., a name of a healthcare professional, a time of theupdate, and the like) that made changes to the report.

In some embodiments, populating a post-operative report of a surgicalprocedure may be configured to cause at least part of derivedimage-based information to be identified in a post-operative report asautomatically generated data. In various embodiments, as derivedimage-based information is used to populate a post-operative report,populating the report may include identifying how the derivedimage-based information was generated. For example, if an elevated heartrate was determined using computer vision analysis of detected pulses invascular, the source of that determination might be noted as being basedon a video determination. Similarly, video analysis might automaticallyestimate a volume of blood loss as the result of a rupture, and thesurgical report might note, along with the estimated loss, that thevolume of loss is an estimation based on video analysis. Indeed, anyindication derived from video analysis might be so noted in thepost-surgical report using any textual, graphical, or icon basedinformation to reflect the source of the data. For example, a movie iconmay appear next to data derived from video. Alternatively, if ahealthcare professional identifies an event within surgical footage andprovides a segment of surgical footage corresponding to the identifiedevent as a derived image-based information, such information may beconsidered as generated by the healthcare professional and may not beclassified as automatically generated data.

Disclosed embodiments may include analyzing surgical footage to identifya surgical event within the surgical footage, for example as describedabove. The analysis, as previously discussed, may occur using a machinelearning model. The identification may be derived from historical datawhere surgical events were already identified, along with a name for theevent. Thus, when a similar even is detected through machine learning,the previously identified name for that event can similarly be appliedto a current event identification.

Further, consistent with disclosed embodiments, not only may an event beidentified, but also a property of a surgical event may also beidentified. The property of a surgical event may be a type of an eventor any other information characterizing the event. For example, if theevent is an incision, the machine-learning model may be configured toreturn a name “incision” as a type of the event, and a length and adepth of the incision as a property of the event. In some cases, apredetermined list of possible types for various events may be providedto a machine-learning model, and the machine-learning model may beconfigured to select a type from the list of event types to accuratelycharacterize an event. The number of properties can vary based on thetype of event identified. Some rather straightforward events may have arelatively short list of associated properties, while other events mayhave many more associated alternative properties.

As discussed, machine-learning models are one way for identifyingevents, with the models trained using examples to identify (ordetermine) events. The training may involve any suitable approach, suchas for example, a supervised learning approach. For instance, historicalsurgical footage containing features corresponding to an event may bepresented as input data for the machine-learning model, and themachine-learning model may output the name of the event corresponding tothe features within the footage. Various parameters of themachine-learning model may be adjusted to train the machine-learningmodel to correctly identify events corresponding to the features withinthe historical visual data. For example, if the machine-learning modelis a neural network, parameters of such a neural network (e.g., weightsof the network, number of neurons, activation functions, biases of thenetwork, number of layers within the network, and the like) may beadjusted using any suitable approach (e.g., weights of the neuralnetwork may be adjusted using a backpropagation process).

In one embodiment, the event may be identified by a medical professional(e.g., a surgeon), and the event may be tagged at the time of itsoccurrence. If a machine learning model identifies surgical activity aspotentially of interest but lacks an associated name for the activity,the associated footage may be saved and a user might later be promptedto provide an associated name.

In some cases, a surgeon may mark an event during a surgical procedurefor subsequent identification. For example, the surgeon may mark theevent using a visual or an audio signal (e.g., a hand gesture, a bodygesture, a visual signal produced by a light source generated by amedical instrument, a spoken word, and the like) that may be captured byone or more image sensors/audio sensors and recognized as a trigger foran event.

In various embodiments, derived image-based information may be based onan identified surgical event and an identified property of the event.After an event and one or more properties of the event are identified asdiscussed earlier, the combination of can be analyzed to determineimage-based information that may not have been derivable from either theevent or the property alone. For example, if a particular property of aparticular event is associated with a known risk of post-operativecomplication, that risk may be determined and included in theimage-based information. Alternatively, by way of example, the derivedimage-based information may include one or more of a name of the event,a segment of a surgical footage corresponding to the event, a nameand/or image of a surgical instrument used during the event, a nameand/or image of an anatomical structure operated during the event, animage of interaction of the surgical instrument and the anatomicalstructure, a duration time for the event, and/or any other informationderived from the video.

As mentioned, the surgical footage may be analyzed to determine an eventname of the identified surgical event. As described above, the eventname may be determined using a suitable machine-learning model.Alternatively, a name of the event may be identified by a healthcareprofessional. In various embodiments, the derived image-basedinformation may include the determined event name.

Aspects of disclosed embodiments may also include associating a timemarker with an identified surgical event. A process of associating atime marker with an identified surgical event may be similar to theprocess of associating a time marker with a phase of a surgicalprocedure. For example, a time marker may be associated with a beginningof an event of a surgical procedure (e.g., the beginning or some otherintermediate location or range of locations of a surgical event withinsurgical footage). A time marker may be any suitable alphanumericalidentifier, or any other graphical or data identifier. For example, thetime marker may be an icon or other graphic that appears on an active orstatic timeline of some or all of a surgical procedure. If active, thetime marker may be clickable (or otherwise selectable) to cause footageof the associated event to be presented. The marker may be caused toappear in footage, either through a textual or graphic overlay on thefootage or through an identifying audio indicator embedded for playbackpresentation. Such indicators may include one or more pieces ofinformation such as temporal data (time or time range of theoccurrence), location data (wherein the event occurred), orcharacterizing data (describing properties of the occurrence.) In somesituations, a time marker may be associated with an end of an event(e.g., the time marker may be associated with an end location of theevent within the surgical footage). Derived image-based information mayinclude multiple time markers, for multiple events and/or for multiplelocations within events.

In some embodiments, providing the derived image-based information mayoccur in a form that enables updating an electronic medical record. Forexample, derived image-based information may include text data, imagedata, video data, audio data, and the like, that may be in a form thatcan be uploaded to a software application that may store and display anelectronic medical record (e.g., a standalone application for storingand displaying a medical record, a web-interface for displaying amedical record using information stored in a database, and the like). Invarious embodiments, the software application for storing and displayinga medical record may include an interface for updating the electronicmedical record using derived image-based information. The interface mayinclude graphical user elements for uploading image, video and audiodata, for uploading text data, for typing text data into the electronicmedical record, for updating the electronic medical record using acomputer mouse, and the like.

In various embodiments, the derived image-based information may be basedin part on a user input. For example, a user, such as a healthcareprofessional, may provide inputs while the surgical footage is beingcaptured, for example as described above, and the derived image-basedinformation may be partly based on such inputs. For example, such inputmay indicate a particular point in time within the surgical footage.

In various embodiments, the derived image-based information may includea first part associated with a first portion of a surgical procedure anda second part associated with a second portion of a surgical procedure.Separating image-based information into parts may facilitate classifyingthe image-based information. For example, if the first portion of thesurgical procedure involves making multiple incisions and a secondportion of the surgical procedure involves suturing, such portions maybe used to classify those portions of the surgical procedure. In somecases, during a first portion of a surgical procedure, a first set ofsensors may be used to collect image-based information, and during asecond portion of the surgical procedure, a different set of sensors maybe used to collect image-based information. For example, during thefirst portion, image sensors located on a surgical instrument may beused to capture surgical footage, and during the second portion of thesurgical procedure, overhead image sensors (i.e., image sensors locatedabove an operating table) may be used to capture the surgical footage.

In various embodiments, the post-operative report may include a firstportion corresponding to the first portion of the surgical procedure anda second portion corresponding to the second portion of the surgicalprocedure. The start of the first portion of the post-operative reportmay be indicated by a first position (e.g., the first position may be apointer in a data file, a location of a cursor in a text file, a datarecord in a database, and the like). The start of the second portion ofthe post-operative report may be indicated by a second position, whichmay be any suitable indication of location in the file that is astarting point of the second portion of the post-operative report (e.g.,the first position may be a pointer in a data file, a location of acursor in a text file, a data record in a database, and the like). Invarious embodiments, a post-operative report may be separated intoportions based on corresponding portions of a surgical procedure. In anexample embodiment, a machine-learning method (or a healthcare provider)may identify portions of the surgical procedure and configure thepost-operative report to have such identified portions. Thepost-operative report may not be limited to two portions but may includemore or less than two portions.

Aspects of disclosed embodiments may include receiving a preliminarypost-operative report. The post-operative report may be received by anyentity, whether an organization, individual, or a computer (e.g., aninsurance company or healthcare organization, a healthcare professional,or a computer-based program for populating post-operative reports, suchas application 2415, as shown in FIG. 24A). In various embodiments,analyzing a preliminary post-operative report may involve selecting afirst position and a second position within the preliminarypost-operative report, the first position is associated with a firstportion of the surgical procedure and the second position is associatedwith a second portion of the surgical procedure. Such selection mayenable someone (or a machine) analyzing the report to skip directly toan area of interest in the report. Thus, analyzing a preliminarypost-operative report may include identifying indicators for one or moreof a first position and a second position. The indicators may be anysuitable alphanumeric or graphical indicators. For example, an indicatorfor the first position may be a text string “this is a start of thefirst portion of the post-operative report” or a graphical start icon.In one example, Natural Language Processing (NLP) algorithms may be usedto analyze textual information included in the preliminarypost-operative report, to identify in the textual information portionsthat discuss different aspects of the surgical procedure (such asdifferent surgical phases, different surgical events, usage of differentmedical instruments, and so forth), and associate the identifiedportions of the textual information with different portions of thesurgical procedure (for example, with the corresponding surgical phase,with the corresponding surgical events, with the usage of thecorresponding medical instruments, and so forth). Further, in someexamples, the first position and the second position (as well asadditional positions) within the preliminary post-operative report maybe based on and/or linked with the identified portions of the textualinformation.

Further, embodiments may include causing a first part of derivedimage-based information to be inserted at a selected first position anda second part of the derived image-based information to be inserted at aselected second position. For example, a first portion of apost-operative report may include a first set of fields that may bepopulated by derived image-based information captured during a firstportion of the surgical procedure, and a second portion of thepost-operative report may include a second set of fields that may bepopulated by derived image-based information captured during a secondportion of the surgical procedure. In another example, a first part ofderived image-based information may correspond to a first portion of thesurgical procedure and a second part of derived image-based informationmay correspond to a second portion of the surgical procedure, the firstposition within the preliminary post-operative report may be identifiedas corresponding to the first portion of the surgical procedure (asdescribed above), the second position within the preliminarypost-operative report may be identified as corresponding to the secondportion of the surgical procedure (as described above), and in response,the first part of derived image-based information may be inserted at thefirst position and the second part of the derived image-basedinformation may be inserted at the second position. Some non-limitingexamples of the first and second portions of the surgical procedure mayinclude different surgical phases, different surgical events, usage ofdifferent medical instruments, different actions, and so forth.

Aspects of the present disclosure may also include analyzing surgicalfootage to select at least part of at least one frame of the surgicalfootage; and causing the selected at least part of at least one frame ofthe surgical footage to be included in a post-operative report of asurgical procedure. For example, if a post-operative report includes afield configured to hold one or more images of a surgical instrumentused during a surgical procedure, an example machine-learning model maybe configured to identify one or more frames of the surgical footage andselect parts of the identified frames that contain a surgicalinstrument. Further, the selected part (or parts) of at least one framemay be inserted (e.g. populate) into the post-operative report. Themachine-learning model may also be configured to extract other relevantframes of surgical footage. For example, frames of the surgical footagedepicting an anatomical structure that is the focus of an operation, orframes depicting an interaction between a surgical instrument and ananatomical structure may be extracted. Such relevant frames may alsopopulate the post-operative report.

Disclosed embodiments may also include receiving a preliminarypost-operative report and analyzing the preliminary post-operativereport and surgical footage to select the at least part of at least oneframe of the surgical footage. For example, a machine-learning model maybe configured to analyze a post-operative report and identify adiscussion of an adverse event (e.g., bleeding). The adverse event maybe identified, for example, through an indication stored in thepost-operative report, using an NLP algorithm, and so forth. Theindication may, for example, be an indication of a name of the adverseevent. It may include a time when the adverse event occurred during asurgical procedure. The adverse event may be determined using amachine-learning model configured to retrieve surgical footage for thesurgical procedure and identify a portion of a frame that shows a visualdata representing the adverse event (e.g., a portion of a frame thatshows bleeding). Further, in some examples, the identify portion of theframe may be inserted to the post-operative report in connection withthe discussion of the adverse event, or be associated with thediscussion of the adverse event in another way.

Additional aspects of disclosed embodiments may include analyzing thepreliminary post-operative report and surgical footage to identify atleast one inconsistency between the preliminary post-operative reportand the surgical footage. In various embodiments, inconsistency may bedetermined by comparing information stored in the report withinformation derived through a machine learning model that determines anerror. For illustrative purposes, one of a virtual infinite number ofpotential inconsistencies could occur when a medical professionalindicates in the report that the surgical site was closed with sutures,while the video reveals that the site was closed with staples. The videorevelation might occur, for example, with a computer-based softwareapplication (e.g., application 2415, as shown in FIG. 24A) where apost-operative report is compared with video footage of the associatedprocedure. If a difference is noted, a computer-based softwareapplication may determine the source of the error, may note the error,may send a notification of the error, and/or may automatically correctthe error. For example, the application may analyze various versions ofa preliminary post-operative report (using, for example, a versiontracking system, as described above) to identify at which step ofgenerating the preliminary post-operative report the difference firstappeared.

As previously mentioned, embodiments of the disclosure may includeproviding an indication of the identified at least one inconsistency.The indication may be provided by transmitting a notification to ahealthcare professional using any suitable means, as discussed above.

Various embodiments may include receiving an input of a patientidentifier and an input of an identifier of a health care provider, aspreviously described. Further, the method may include receiving an inputof surgical footage of a surgical procedure performed on the patient bythe health care provider, as previously described. The method may alsoinclude analyzing a plurality of frames of the surgical footage toidentify phases of the surgical procedure based on detected interactionsbetween medical instruments and biological structures and, based on theinteractions, associate a name with each identified phase. For example,at least some of the frames of the surgical footage may indicate aportion of the surgical footage in which a surgical operation is beingperformed on a biological structure (herein, also referred to as ananatomical structure). As discussed above, the interaction may includeany action by the medical instrument that may influence the biologicalstructure or vice versa. For example, the interaction may include acontact between the medical instrument and the biological structure, anaction by the medical instrument on the biological structure (such ascutting, clamping, grasping, applying pressure, scraping, etc.), aphysiological response by the biological structure, the medicalinstrument emitting light towards the biological structure (e.g.,surgical tool may be a laser that emits light towards the biologicalstructure) a sound emitted towards anatomical structure, anelectromagnetic field created in a proximity of the biologicalstructure, a current induced into the biological structure, or any othersuitable forms of interaction.

In some cases, detecting an interaction may include identifyingproximity of the medical instrument to a biological structure. Forexample, by analyzing the surgical video footage, an image recognitionmodel may be configured to determine a distance between the medicalinstrument and a point (or a set of points) on a biological structure.

Aspects of the present disclosure may involve associating a name witheach identified phase based on detected interactions between medicalinstruments and biological structures. The name may be associated witheach identified phase using any suitable means. For example, asdescribed above, the name may be supplied by a user or may beautomatically determined using a suitable machine learning method, asdescribed above. In particular, a process of identifying a phase of asurgical procedure involves associating a name with each identifiedphase. In various embodiments, the name associated with the phase mayinclude a name for a biological structure and a name of a surgicalinstrument interacting with the structure.

In various embodiments, the name associated with the identified phasemay be updated, modified, quantified, or otherwise altered during theongoing surgical phase or after the completion of the surgical phase.For example, a machine learning model may initially determine a name forthe surgical phase as “incision” and may later update the name of thesurgical phase, based on detected interactions between medicalinstruments and biological structures, to an illustrative name “a Lanzincision extending medially towards rectus abdominis, made vialaparoscopic surgery using laparoscopic scissors.” Additionally oralternatively, a separate record (herein also referred to as a note) maybe added to the name identifying the surgical phase, with the notecontaining various details and/or characteristics of the surgical phase.Such details may include an instrument used during the surgical phase, alight used during the surgical phase, a pressure value for the pressureapplied on an example biological structure, an area over which thepressure was applied, one or more images of the biological structureand/or medical instrument during the surgical phase, identifications forevents (e.g., adverse events such as bleeding), or any other relatedinformation characterizing the surgical phase.

Aspects of the present disclosure may also involve transmitting data toa health care provider, the transmitted data including the patientidentifier, the names of the identified phases of the surgicalprocedure, and time markers associated with the identified phases.

An embodiment may include determining at least a beginning of eachidentified phase, and associating a time marker with the beginning ofeach identified phase, as discussed above. Additionally oralternatively, the time marker may identify an end of the identifiedphase, as discussed above. The transmitted data may include text,graphics, video data, animations, audio data, and the like. In somecases, the transmitted data may be an SMS message, an email, and thelike delivered to any suitable devices (e.g., smartphones, laptops,desktops, TVs, etc.) in possession of various health care providers(e.g., various medical personnel, administrators, and other interestedindividuals or systems). In some cases, the transmitted data may also beprovided to patients, relatives or friends of patients.

Further, aspects of the present disclosure may include populating apost-operative report with transmitted data in a manner that enables thehealth care provider to alter phase names in a post-operative report.Such alterations may occur through an interface that enablespost-operative report alterations. For example, the interface may allowa healthcare provider to update the phase names by typing new phasenames using a keyboard. In various embodiments, the interface may bealso configured for altering names of various events identified insurgical footage and recorded in a post-operative report.

Disclosed systems and methods may involve analyzing surgical footage toidentify events during the surgical procedure, comparing the events witha sequence of recommended events, and determining if any events from thesequence of the recommended events were not performed during thesurgical procedure. Omitted surgical events may need to be identifiedduring or after a surgical procedure. The events may be compared with asequence of recommended events, and when some events were not performedduring the surgical procedure, as determined by comparing with thesequence of recommended events, a notification may be provided toindicate which event has been omitted. Therefore, there is a need foranalyzing surgical footage and identifying omitted events during asurgical procedure.

Aspects of this disclosure may relate to enabling determination andnotification of an omitted event in a surgical procedure, includingrelated methods, systems, devices, and computer readable media.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In the broadest sense, the method is not limitedto particular physical and/or electronic instrument, but rather may beaccomplished using many differing instruments.

Disclosed embodiments may include enabling determination andnotification of an omitted event may involve accessing frames of videocaptured during a specific surgical procedure. As used herein, frames ofthe video may include sequential or non-sequential images captured by animage capture device. Such images may be captured by, for example,cameras 115, 121, 123, and/or 125, as described above in connection withFIG. 1 . In some cases, frames of the video may have corresponding audiosignals forming a soundtrack for the video, with the audio signals beingcaptured by audio capturing devices (e.g., microphone D111, as shown inFIG. 1 ). The video frames may be stored as individual files or may bestored in a combined format, such as a video file, which may includecorresponding audio data. In some embodiments, a video may be stored asraw data and/or images output from an image capture device. In otherembodiments, the video frames may be processed. For example, video filesmay include Audio Video Interleave (AVI), Flash Video Format (FLV).QuickTime File Format (MOV), MPEG (MPG, MP4, M4P, etc.), Windows MediaVideo (WMV), Material Exchange Format (MXF), a non-compressed videofile, a lossless compressed video file, a lossy compressed video file,or any other suitable video file formats.

A specific surgical procedure, as used herein, may include any medicalaction, operation, diagnosis, or other medical related procedure oraction. Such procedures may include cutting, ablating, suturing, orother techniques that involve physically changing body tissues andorgans. Some examples of such surgical procedures may include alaparoscopic surgery, a thoracoscopic procedure, a bronchoscopicprocedure, a microscopic procedure, an open surgery, a robotic surgery,an appendectomy, a carotid endarterectomy, a carpal tunnel release, acataract surgery, a cesarean section, a cholecystectomy, a colectomy(such as a partial colectomy, a total colectomy, etc.), a coronaryartery bypass, a debridement (for example of a wound, a burn, aninfection, etc.), a free skin graft, a hemorrhoidectomy, a hipreplacement, a hysteroscopy, an inguinal hernia repair, a sleevegastrectomy, a ventral hernia repair, a knee arthroscopy, a kneereplacement, a mastectomy (such as a partial mastectomy, a totalmastectomy, a modified radical mastectomy, etc.), a prostate resection,a prostate removal, a shoulder arthroscopy, a spine surgery (such as aspinal fusion, a laminectomy, a foraminotomy, a diskectomy, a diskreplacement, an interlaminar implant, etc.), a tonsillectomy, a cochlearimplant procedure, brain tumor (for example meningioma, etc.) resection,interventional procedures such as percutaneous transluminal coronaryangioplasty, transcatheter aortic valve replacement, minimally invasivesurgery for intracerebral hemorrhage evacuation, thoracoscopicprocedure, bronchoscopy, hernia repair, hysterectomy (e.g., a simplehysterectomy, or a radical hysterectomy), radical prostatectomy, partialnephrectomy, thyroidectomy, hemicolectomy, or any other medicalprocedure involving some form of incision, diagnosis, treatment ortissue alteration, or involving for example, treatment, diagnosis, drugadministration, excision, repair, implantation, reconstruction, orimprovement.

A deviation between a specific surgical procedure and a recommendedsequence of events may be specific to a surgical procedure, as each typeof surgical procedure may involve one or more of its own recommendedsequences of events. When one such recommended sequence is not followed,a deviation may be said to have occurred, and a notification may beprovided (for example as described below). In some gallbladder surgeries(such as a laparoscopic or a robotic cholecystectomy), for example, adeviation may include neglecting to clear a hepatocytic triangle of fatand fibrous tissue, to separate a gallbladder from a liver, to expose acystic plate, or a failure to identify a cystic duct and a cystic arteryentering a gallbladder. By way of another example, in some appendixsurgeries (such as a laparoscopic or a robotic appendectomy), adeviation may include neglecting to dissect an appendix from surroundingadhesions or may include a failure to identify a base on an appendixcircumferentially. In some hernia surgeries (such as a laparoscopicventral hernia repair), a deviation may include neglecting to reducehernia content, neglecting to visualize the fascia surrounding thehernia before anchoring a mesh, neglecting to isolate a fasciasurrounding the hernia or neglecting to identify and/or isolate aninguinal canal element, and so forth. An example of such inguinal canalelement may be a testicular artery, a pampiniform plexus of veins,nerves, a vas, and so forth. In some uterine surgery, such as alaparoscopic simple hysterectomy, a deviation may include neglecting toidentify and/or ligate uterine arteries, neglecting to identify ureters,and so forth. In some other uterine surgeries, such as a robotic radicalhysterectomy, a deviation may include neglecting to identify iliac bloodvessels, neglecting to identify an obturator nerve, and so forth. Insome prostate surgeries, such as a robotic radical prostatectomy, adeviation may include neglecting to identify a bladder neck in ananterior bladder wall, neglecting to identify a bladder neck in aposterior bladder wall, neglecting to identify ureteral orifices, and/orneglecting to identify other anatomical structures. In proceduresinvolving the kidney, such as a laparoscopic or a robotic partialnephrectomy, the deviation may include neglecting to identify a renalhilum, where neglecting to identify the renal hilum may includeneglecting to identify at least one of an artery, a vein, and collectingsystem including a ureter. In thyroid surgery, such as an open or arobotic thyroidectomy, a deviation may include neglecting to identify arecurrent laryngeal nerve. In colon procedures (such as a colectomy or ahemicolectomy, whether open, laparoscopic or robotic), a deviation mayinclude neglecting to dissect a colon from a retroperitoneum, neglectingto dissect a colon from a liver, neglecting to dissect a colon fromsplenic flexures, or neglecting to perform an anastomosis, neglecting tovisualize a colon free from adhesions and/or with no tension, neglectingto perform anastomosis, neglecting to visualize a tension free and/orwell perfused and/or technically well sealed anastomosis, and so forth.The forgoing are just a few examples. More broadly, any divergence froman expected or recognized course of action may be considered adeviation.

A surgical procedure may take place in an operating room or any othersuitable location. An operating room may be a facility (e.g., a roomwithin a hospital) where surgical operations are carried out in anaseptic environment. The operating room may be configured to be well-litand to have overhead surgical lights. The operating room may featurecontrolled temperature and humidity and may be windowless. In anexemplary embodiment, the operating room may include air handlers thatfilter the air and maintain a slightly elevated pressure within theoperating room to prevent contamination. The operating room may includean electricity backup system in case of a black-out and may include asupply of oxygen and anesthetic gases. The room may include a storagespace for common surgical supplies, containers for disposables, ananesthesia cart, an operating table, cameras, monitors, and other itemsfor surgery. A dedicated scrubbing area that is used by surgeons,anesthetists, operating department practitioners (ODPs), and nursesprior to surgery may be part of the operating room. Additionally, a mapincluded in the operating room may enable the terminal cleaner torealign the operating table and equipment to the desired layout duringcleaning. In various embodiments, one or more operating rooms may be apart of an operating suite that may form a distinct section within ahealthcare facility. The operating suite may include one or morewashrooms, preparation and recovery rooms, storage and cleaningfacilities, offices, dedicated corridors, and possibly other supportiveunits. In various embodiments, the operating suite may be climate-and/or air-controlled and separated from other departments.

Accessing the video frames of video captured during a specific surgicalprocedure may include receiving the frames from an image sensor (ormultiple image sensors) located in an operating room. An image sensormay be any detector capable of capturing image or video data. A videoframe may include at least a portion of one of many still images thatcompose a moving picture, such as a clip of any duration. Capturing ofvideo may occur when one or more still images or portions thereof arereceived from an image sensor. Alternatively or additionally, capturemay occur when one or more still images or portions thereof areretrieved from memory in a storage location. For example, video framesmay be accessed from a local memory, such as a local hard drive, or maybe accessed from a remote source, for example, through a networkconnection. In an example embodiment, the video frames may be retrievedfrom database 1411, as shown in FIG. 14 . For example, processor 1412 ofsystem 1410 may be configured to execute instructions (e.g.,instructions implemented as software 1416) to retrieve the video framesfrom database 1411. The video frames may be retrieved for a specificsurgical procedure.

Aspects of embodiments for enabling determination and notification of anomitted event may further include accessing stored data identifying arecommended sequence of events for the surgical procedure. As usedherein, an event for the surgical procedure (also referred to as asurgical event) may refer to an action that is performed as part of asurgical procedure (e.g., an intraoperative surgical event), such as anaction performed by a surgeon, a surgical technician, a nurse, aphysician's assistant, an anesthesiologist, a doctor, or any otherhealthcare professional. An intraoperative surgical event may be aplanned event, such as an incision, administration of a drug, usage of asurgical instrument, an excision, a resection, a ligation, a graft,suturing, stitching, or any other planned event associated with asurgical procedure or phase.

An example of a surgical event in a laparoscopic cholecystectomy surgerymay include trocar placement, calot's triangle dissection, clipping andcutting of cystic duct and artery, gallbladder dissection, gallbladderpackaging, cleaning and coagulation of liver bed, gallbladderretraction, and so forth. In another example, surgical events of acataract surgery may include povidone-iodine injection, cornealincision, capsulorhexis, phaco-emulsification, cortical aspiration,intraocularlens implantation, intraocular-lens adjustment, woundsealing, and so forth. In yet another example, surgical characteristicevents of a pituitary surgery may include preparation, nasal incision,nose retractor installation, access to the tumor, tumor removal, columnof nose replacement, suturing, nose compress installation, and so forth.The foregoing are just a few examples to illustrate the distinctionbetween a surgical procedure and an event within the surgical procedureand are not intended to be limiting of the embodiments described herein.Some other examples of common surgical events may include incisions,laparoscope positioning, suturing, and so forth.

In some embodiments, the surgical event may include an unplanned event,an adverse event or a complication. Some examples of adverse surgicalevents may include bleeding, mesenteric emphysema, injury, conversion tounplanned open surgery (for example, abdominal wall incision), incisionsignificantly larger than planned, and so forth. Some examples ofintraoperative complications may include hypertension, hypotension,bradycardia, hypoxemia, adhesions, hernias, atypical anatomy, duraltears, periorator injury, arterial occlusions, and so forth. In somecases, surgical events may include other errors, including technicalerrors, communication errors, management errors, judgment errors,situation awareness errors, decision-making errors, errors related tomedical equipment utilization, and so forth. In various embodiments,events may be short or may last for a duration of time. For example, ashort event (e.g., incision) may be determined to occur at a particulartime during the surgical procedure, and an extended event (e.g.,bleeding) may be determined to occur over a time span. In some cases,extended events may include a well-defined beginning event and awell-defined ending event (e.g., beginning of suturing and ending of thesuturing), with suturing being an extended event. In some cases,extended events are also referred to as phases during a surgicalprocedure.

In various embodiments, a recommended event may be an event that isrequired during a surgical procedure. Similarly, a recommended event maybe an event that is suggested to occur during a surgical procedure. Forexample, a recommended event during bronchoscopy may include insertionof a bronchoscope through a patient's nose or mouth, down the patient'sthroat into the patient's lungs. A recommended sequence of events mayinclude a recommended sequence of recommended events. In some cases, asurgical event may identify a group of sub-events (i.e., more than onesub-event or steps). For example, an event of administering generalanesthesia to a patient may include several steps such as a first stepof providing medication to a patient via an IV line to induceunconsciousness, and a second step of administering a suitable gas(e.g., isoflurane or desflurane) to maintain the general anesthesia.

In an example embodiment, a recommended event may include administeringa patient a pain-relief medicine, placing a patient in a preferredposition, obtaining a biopsy sample from the patient, or any othersuggested event that is not required.

The recommended sequence of events may include any suitable establishedsequence of events used during a surgical procedure. The recommendedsequence of events may be established by healthcare professionals (e.g.,surgeons, anesthesiologists, or other healthcare professionals) byanalyzing historical surgical procedures and determining guidelines forsurgical procedures. Examples of the recommended sequence of events mayinclude, for example, inspecting an appendix base in a circumferentialview. In some cases, the recommended sequence of events may be based ona critical view of safety (CVS), as known in the art. For example,during a laparoscopic cholecystectomy critical view of safety may beused to identify a cystic duct and a cystic artery to minimize injuriesto a bile duct. In other embodiments, a determination of mandatory andrecommended sequences of events may be determined automatically throughthe application of artificial intelligence to historical surgical videofootage.

By way of illustration, in some embodiments, a CVS may be used to avoidbiliary injury. The CVS may be used to identify the two tubularstructures that are divided in a cholecystectomy, i.e., the cystic ductand the cystic artery. The CVS may be used as a process in an opencholecystectomy in which both cystic structures are putativelyidentified, after which the gallbladder is taken off the cystic plate sothat it is hanging free and attached by the two cystic structures. Inlaparoscopic surgery, a complete separation of the body of thegallbladder from the cystic plate makes clipping of the cysticstructures difficult. Thus, for the laparoscopy, the requirement may bethat a lower part of the gallbladder (about one-third) may be separatedfrom the cystic plate. The other two requirements may be that thehepatocytic triangle is cleared of fat and fibrous tissue and that thereare two and only two structures attached to the gallbladder. Not untilall three elements of CVS are attained, may the cystic structures beclipped and divided. Intraoperatively CVS should be confirmed in a“time-out” in which the three elements of CVS are demonstrated. Itshould be noted that CVS is not a method of dissection but a method oftarget identification akin to concepts used in safe hunting procedures.

The recommended sequence of events may include conditional clauses. Asan illustrative example, recommended sequence of events for bypasssurgery may include (1) administering general anesthesia for a patient,(2) preparing the arteries that will be used as bypass grafts, (3)making an incision at the center of a patient's chest, through a sternum(breast bone), to access heart and coronary arteries of the patient. (4)connecting a heart-lung bypass machine, (5) sewing one section of theartery around an opening below the blockage in the diseased coronaryartery while a patient's heart is beating, (6) checking if the patient'sheart continues to pump blood, (7) if the patient's heart stops beatingactivate the heart-lung bypass machine, (8) attaching the other end toan opening made in the aorta, and the like. As described above, theevent of activating the heart-lung bypass machine may be part of therecommended sequence of events and may be triggered by any suitablesurgical events (or lack of thereof), such as a surgical event ofcessation of heart beats. In some cases, the recommended sequence ofevents may include a decision tree for determining the next event in thesequence of events. In some examples, the recommended sequence of eventsmay include events that are required to occur within a particular timeinterval that may be specified in the recommended sequence of events.For example, an event may be required to occur within a particular timeinterval of the surgical procedure, within a particular time intervalafter the beginning of the surgical procedure, within a particular timeinterval before the completion of a surgical procedure, within aparticular time interval of the surgical procedure after an occurrenceof a second event (e.g., after the completion of the second event, afterthe beginning of the second event, etc.), within a particular timeinterval of the surgical procedure before an occurrence of a secondevent, and so forth.

Accessing the stored data identifying a recommended sequence of eventsmay include retrieving the stored data from a suitable storage location(e.g., a data storage device such as a memory, a hard drive, a database,a server, and the like). In an example embodiment, the stored data maybe retrieved from database 1411, as shown in FIG. 14 . For example,processor 1412 of system 1410 may be configured to execute instructions(e.g., instructions implemented as software 1416) to retrieve storeddata from database 1411. The stored data may be retrieved for a specificsurgical procedure. In some examples, identifying a recommended sequenceof events may include selecting the recommended sequence of events of aplurality of alternative sequences. For example, the recommendedsequence of events may be selected based on the type of the surgicalprocedure, based on a medical instrument being used or projected to beused in the surgical procedure, based on a condition of an anatomicalstructure related to the surgical procedure, based on characteristics ofa patient associated with the surgical procedure (some examples of suchcharacteristics are described above), based on characteristics of asurgeon or a medical care professional associated with the surgicalprocedure (some examples of such characteristics are described above),based on characteristics of an operating room associated with thesurgical procedure, and so forth. In some examples, the recommendedsequence of events may be selected (or modified) during a surgicalprocedure according to one or more events that already occurred in thesurgical procedure. For example, an occurrence of a particular event ina surgical procedure may indicate a type of the surgical procedure (forexample, a location and/or a length of an incision may indicate whetherthe surgical procedure is an open surgical procedure or a laparoscopicsurgical procedure, a usage of a particular medical instrument mayindicate an election of a particular technique which may requireparticular sequence of events, etc.) or a technique that a surgeonelected for the particular surgical procedure, and a correspondingrecommended sequence of events may be selected. In another example, anoccurrence of a particular event in a surgical procedure may indicate acomplication that necessitates a different recommended sequence ofevents, and a corresponding sequence of events may be selected. In yetanother example, in response to a first event occurring in a particularongoing surgical procedure, a first recommended sequence of events maybe selected for a remaining portion of the particular ongoing surgicalprocedure, and in response to a second event occurring in a particularongoing surgical procedure, a second recommended sequence of events maybe selected for the remaining portion of the particular ongoing surgicalprocedure, the second recommended sequence of events may differ from thefirst recommended sequence of events. In some examples, image datacaptured from a particular ongoing surgical procedure may be analyzed toselect a recommended sequence of events for a remaining portion of theparticular ongoing surgical procedure. For example, the image data maybe analyzed to detect events and/or conditions in the particular ongoingsurgical procedure (for example, as described above), and therecommended sequence of events may be selected based on the detectedevents and/or conditions. In another example, a machine learning modelmay be trained using training examples to select recommended sequence ofevents based on images and/or videos, and the trained machine learningmodel may be used to analyze the image data and select the recommendedsequence of events for a remaining portion of the particular ongoingsurgical procedure. An example of such training example may include animage and/or a video depicting a first part of a surgical procedure,together with a label indicating a desired selection of a recommendedsequence of events for a remaining part of the surgical procedure.

An example recommended sequence of events 2601 is schematicallyillustrated in FIG. 26 . For example, an event E1 (e.g., connecting aheart-lung bypass machine) may be a first event in the recommendedsequence. Event E1 may be required to occur during a time intervalT1A-T1B of the surgical procedure. An event E2 (e.g., suturing), may bea second event and may be required to occur during a time intervalT2A-T2B of the surgical procedure (or in other examples, during a timeinterval T2A-T2B after the completion of event E1, during a timeinterval T2A-T2B after the beginning of event E1, and so forth). Aftercompletion of event E2, a conditional statement C1 (e.g., determining apulse of a patient's heart) may be evaluated. If conditional statementC1 evaluates to value V1 (e.g., if the patient has no pulse), an eventE3 (e.g., activate the heart-lung bypass machine) may be required duringa time interval T3A-T3B. If statement C1 evaluates to value V2 (e.g.,pulse of ten beats per minute) an event E4 (e.g., administer a firstmedicine to the patient) may be required during a time interval T4A-T4B,and if statement C1 evaluates to value V3 (e.g., pulse of hundred beatsper minute) an event E5 (e.g., administer a second medicine to thepatient) may be required during a time interval T5A-T5B.

Aspects of the method for enabling determination and notification of theomitted event may further include comparing the accessed video frameswith the recommended sequence of events to identify an indication of adeviation between the specific surgical procedure and the recommendedsequence of events for the surgical procedure. In some examples, amachine learning model may be trained using training examples toidentify indications of deviations between the surgical procedures andrecommended sequence of events for the surgical procedures from imagesand/or videos, and the trained machine learning model may be used toanalyze the video frames and identify the indication of the deviationbetween the specific surgical procedure and the recommended sequence ofevents for the surgical procedure. An example of such training examplemay include a sequence of events and images and/or videos depicting asurgical procedure, together with a label indicating whether thesurgical procedure deviated from the sequence of events.

In some examples, comparing the accessed video frames with therecommended sequence of events may include analyzing the video framesand identifying events within the video frames, for example as describedabove. For example, identifying events within the video frames may beaccomplished using a trained machine-learning model, for example asdescribed above. In one example, identifying an event may include atleast one of identifying a type of the event, identifying a name of theevent, identifying properties of the event (some examples of suchproperties are described above), identifying an occurrence time (or atime interval) of the event, and so forth. Further, in some examples,the identified events may be compared with the recommended sequence ofevents to identify the indication of the deviation between the specificsurgical procedure and the recommended sequence of events for thesurgical procedure. In some examples, the analysis of the video framesand the identification of the events within the video frames mayoccurred while the specific surgical procedure is ongoing, and thedeviation between the specific surgical procedure and the recommendedsequence of events for the surgical procedure may be identified whilethe specific surgical procedure is ongoing. In other examples, theanalysis of the video frames and the identification of the events withinthe video frames may occurred after a completion of the specificsurgical procedure, and/or the deviation between the specific surgicalprocedure and the recommended sequence of events for the surgicalprocedure may be identified after the specific surgical procedure iscompleted.

Detecting a characteristic event using a machine-learning method may beone possible approach. Additionally or alternatively, the characteristicevent may be detected in the video frames received from image sensorsusing various other approaches. In one embodiment, the characteristicevent may be identified by a medical professional (e.g., a surgeon)during the surgical procedure. For example, the characteristic event maybe identified using a visual or an audio signal from the surgeon (e.g.,a hand gesture, a body gesture, a visual signal produced by a lightsource generated by a medical instrument, a spoken word, and the like)that may be captured by one or more image sensors/audio sensors andrecognized as a trigger for the characteristic event.

Further, comparing the accessed video frames with the recommendedsequence of events may include comparing a sequence of the identifiedevents within the video frames with the recommended sequence of eventsfor the surgical procedure. For example, FIG. 27 shows a sequence 2701of recommended (or mandatory) events and a sequence 2702 of theidentified events within the video frames. When comparing sequence 2701with sequence 2702, a deviation of sequence 2702 from sequence 2701 maybe determined. Sequence 2702 may deviate from sequence 2701 in a varietyof ways. In some cases, sequence 2702 may have different events thansequence 2701. For example, sequence 2701, as shown in FIG. 27 may haveevents E1-E4, and sequence 2702 may have events S1-S5. Sequences 2701and 2702 may be compared for each of intervals I1-I4, as shown in FIG.27 . For example, event E1 of sequence 2701 may be compared with eventS1 for interval I1 of the sequences. In an example embodiment, event E1may deviate from event S1. Alternatively, event E1 may be substantiallythe same as event S1. In some cases, event E1 may be substantiallydifferent from event S1.

In various embodiments, to quantify a difference between event E1 andevent S1, a suitable measure function F(E1,S1) may be defined that mayhave a range of values. In an example embodiment, measure function F mayreturn a single number that determines a difference between events E1and S1. For instance, if F(E1, S1)<F₀(E1), events E1 and S1 aredetermined to be substantially the same, whereas if F(E1,S1)>F₀(E1),events E1 and S1 are determined to be substantially different. Herein,values F₀ and F₁ may be any suitable predetermined threshold values,which may be selected for each type of event (i.e., threshold valuesF₀(E1) and F₁(E1) for event E1 may be different from threshold valuesF₀(E2) and F₁(E₂) for event E2). In various cases, events E1 and S1 maybe characterized by a set of parameters (also referred to as eventcharacteristics). For example, event E1 may be characterized byparameters P1 _(E1)-PN_(E1), as shown in FIG. 27 . Parameters P1_(E1)-PN_(E1) may include words, numbers, or data that may berepresented by an array of numbers (e.g., images). For instance,parameter P1 _(E1) may indicate a type of event E1 characterized by atext string (e.g., “incision”), parameter P2 _(E1) may be a numbercharacterizing a length of the incision (e.g., one centimeter),parameter P3 _(E1) may be the depth of the incision (e.g., threemillimeters), parameter P4 _(E1) may be a location of the incision thatmay be characterized by two numbers (e.g., {10,20}). The location ofincision may be specified by identifying the incision in one or more ofthe video frames captured during the surgical procedure, and parameterPN_(E1) may indicate a type of surgical tool used for the incision(e.g., “CO2 laser”). Event E1 may have as many parameters as needed tofully characterize the event. Further event E1 may be characterized by astarting time TS_(E1) and a finishing time TF_(E1) which may be definedto any suitable precision (e.g., to a precision of a millisecond).TS_(E1) and TF_(E1) may be represented using any suitable time format(e.g., the format may be hour:minute:second:millisecond). Similarly,event S1 may be characterized by parameters P1 _(S1)-PN_(S1), startingtime TS_(S1), and a finishing time TF_(S1), as shown in FIG. 27 . As anillustrative example, parameters {P1 _(E1), P2 _(E1), P3 _(E1), P4_(E1), PN_(E1), TS_(E1), TF_(E1)} may be represented by any suitabledata structure (e.g., {P1 _(E1), P2 _(E1), P3 _(E1), P4 _(E1), PN_(E1),TS_(E1), TF_(E1)}={“incision”, 1 [cm], 3 [mm], {10,20}, “CO2 laser”,13:20:54:80, 13:20:59:78}).

In various embodiments, measure function F(E1,S1) may be defined in anysuitable way. As an example embodiment, measure function may be definedas F(E1,S1)=Σ_(i)(P_(l) _(E1) −P_(l) _(S1) )²+Σ_(k) M(P_(k) _(E1) −P_(k)_(S1) ), where P_(l) _(E1) and P_(l) _(S1) are related numericalparameters, when event E1 and event S1 are of the same type (e.g., bothevents are of type “incision”), where parameters P_(k) _(E1) , and P_(k)_(S1) are text strings (or data, such as images, that may be representedby arrays of numbers), and where function M returns zero if text stringsP_(k) _(E1) , and P_(k) _(S1) contain the same meaning, or returns oneif text strings P_(k) _(E1) , and P_(k) _(S1) contains a differentmeaning. For cases when P_(k) _(E1) , and P_(k) _(S1) correspond toimages, function M may return zero if images are substantially the sameor return one if images are different. In various embodiments, theimages may be compared using any suitable image recognition algorithmfurther described below. Alternatively, function M may be configured toexecute any suitable algorithm for comparing P_(k) _(E1) , and P_(k)_(S1) depending on a type of data represented by parameters P_(k) _(E1)and P_(k) _(S1) , where the data may include text strings, an array ofnumbers, images, videos, audio signals, and the like.

For cases when events E1 and S1 are not of the same type (e.g., event E1may correspond to “incision” and event S1 may correspond to“administering a medication”), and when sequence 2702 does not containan event of the same type as event E1, the measure function F(E1,S1) maybe evaluated to a large predetermined number (or string) indicating thatevents E1 and S1 are substantially different.

As described above the deviation between sequence of events 2701 and2702 may be determined by evaluating a suitable measure functionF(E_(i),S_(i)) for each interval of a surgical procedure I1-I4. Acomplete deviation may be calculated as a sum of measure functionsΣ_(i)(E_(i),S_(i)), where i={I1 . . . I4}. In various embodiments,however, calculating all the deviations for all of the events S1-S4 fromthe corresponding events E1-E4 may not be important and/or necessary. Invarious cases only large deviations (i.e., deviations whereF(E_(i),S_(i))>F₁(E_(i)) may be important. For such deviations, eventsE_(i), S_(i) may be identified and stored for further analysis.Additionally, a value of measure function F(E_(i),S_(i)) may be storedfor further analysis as well. In various embodiments, data related toevents E_(i), S_(i), and measure function F(E_(i),S_(i)) may be storedusing any suitable means (e.g., hard drive, database 111, and the like).

Using a measure function may be one possible approach of identifying anindication of a deviation between the specific surgical procedure andthe recommended sequence of events for the surgical procedure. Forexample, any algorithm for comparing lists and/or graphs may be used tocompare the actual sequence of events with the recommended sequence ofevents and to identify an indication of a deviation between the specificsurgical procedure and the recommended sequence of events for thesurgical procedure. Alternatively or additionally, identifying anindication of a deviation occurs using a machine learning model trainedusing training examples to identify indications of deviations between asequence of events and surgical footage, for example as described above.In an example embodiment, an illustrative training example may includesurgical footage such as frames of a video captured during a surgicalprocedure of a particular type (e.g., cholecystectomy), as well as therecommended sequence of events for that type of surgical procedure. Thetraining example may be used as an input for the machine-learningtraining algorithm, and the resulting machine learning model may be asuitable measure of deviation between the specific surgical procedureand the recommended sequence of events for the surgical procedure. Themeasure of deviation may be any suitable measure. In an exampleembodiment, the measure may list or classify events during the surgicalprocedure, which are substantially different from the recommendedevents. For example, if a recommended event requires suturing, butsurgical glue was used instead during the surgical procedure, such anevent may be listed or classified as substantially different from therecommended event. Additionally or alternatively, the measure may listrecommended events that were not performed during the surgical procedure(e.g., if suturing was required but not performed, such event may belisted as not being performed). Furthermore, the measure may list eventsduring the surgical procedure that were performed but are notrecommended events. For example, an event of administering apain-relieving medicine to a patient during the surgical procedure maybe performed and may not be recommended. Additionally, themachine-learning model may output deviations between characteristics ofevents performed during the surgery and the corresponding recommendedevents, as described above. For example, if during an incision eventduring the surgical procedure, the incision length is shorter than anincision described by the recommended event, such deviation may beidentified by the machine-learning method and recorded (e.g., stored)for further analysis.

In various embodiments, identifying an indication of a deviationincludes comparing the frames to reference frames depicting therecommended sequence of events. The reference frames may be historicalframes captured during historical surgical procedures. In an exampleembodiment, the video frames and the reference frames depicting therecommended sequence of events may be synchronized by an event (hereinalso referred to as a starting event) that may be the same (orsubstantially similar) as a corresponding starting event of therecommended (or mandatory) sequence of events. In some cases, a framedepicting the beginning of the starting event may be synchronized with areference frame depicting the starting event of the recommended sequenceof events. In some cases, events of the surgical procedure may be firstcorrelated to corresponding reference events of the recommendedsequence, using any suitable approaches described above (e.g., using animage recognition algorithm for recognizing events). After correlatingan example surgical event with a corresponding reference event of therecommended sequence, a frame depicting the start of the surgical eventmay be synchronized with a reference frame depicting the start of thecorresponding recommended event.

Additionally or alternatively, identifying an indication of a deviationmay be based on an elapsed time associated with an intraoperativesurgical procedure. For example, if the elapsed time associated with thesurgical procedure is significantly longer (or shorter) than an averageelapsed time associated with the surgical procedure, having arecommended sequence of events, the method may be configured todetermine that the deviation from the recommended sequence of events hasoccurred.

Aspects of the method may also include identifying a set of frames ofthe surgical procedure associated with the deviation and providing thenotification that the deviation has occurred. The notification mayinclude displaying the identified set of frames associated with thedeviation. For example, the set of frames associated with the deviationmay depict a particular event during the surgical procedure that isdifferent (e.g., have different characteristics) than a referencecorresponding recommended event. Alternatively, the set of framesassociated with the deviation may include frames for an event that isnot present in the recommended sequence of events. In variousembodiments, the notification may include displaying the frames as stillimages or displaying the frames as video data. The frames may bedisplayed on any suitable screen of an electronic device or (in somecases) may be printed. In some embodiments, some of the frames may beselected from the set of frames and displayed using any suitable means(e.g., using display screens of electronic devices).

Aspects of the method for enabling determination and notification of theomitted event may further include training the machine learning modelusing the training examples to identify deviations between a sequence ofevents and surgical footage, for example as described above. Forexample, training examples may be used as an input for themachine-learning model, and the measure of the deviation returned by themodel may be analyzed (e.g., the measure of the deviation may beanalyzed by a model training specialist, such as a healthcareprofessional). If the measure of the deviation returned by the modeldoes not coincide with a desired measure of the deviation, variousparameters of the machine-learning model may be adjusted to train themachine-learning model to correctly predict the measure of thedeviation. For example, if the machine-learning model is a neuralnetwork, parameters of such a neural network (e.g., weights of thenetwork, number of neurons, activation functions, biases of the network,number of layers within the network, and the like) may be adjusted usingany suitable approach (e.g., weights of the neural network may beadjusted using a backpropagation process). In various embodiments, suchadjustments may be made automatically (e.g., using the backpropagationprocess), or in some cases, adjustments may be made by the trainingspecialist.

In various embodiments, how well the measure of the deviation coincideswith the desired measure of the deviation may be asserted using anysuitable, appropriate mathematical measure function G. For example, if ameasure of a deviation for an event is a number, (e.g., d), and thedesired measure of the deviation is another number (e.g., d₀) then anexample mathematical measure function for a given event E; may beG_(i)(d,d₀) may be G_(i)(d,d₀)=d−d₀, and the measure function may be,for example, a number G=Σ_(i) G_(i)(d_(i),d_(i) ₀ )². Alternatively, inanother example embodiment, G may be a vector G={G_(i)(d_(i),d_(i) ₀ )}.

To further illustrate a process of determining the deviation of sequence2702 from sequence 2701, FIG. 27 shows intervals 11-14 at which eventsE1-E4 of sequence 2701 may be compared with events S1-S5 of sequence2702. For example, during interval 11, event S1 may be substantially thesame as event E1, and during interval 12 event S2 may deviate from eventE2 but may be sufficiently similar to event E2. For example, event S2may correspond to “incision” having an incision length of threecentimeters, and event E2 may correspond to “incision” having anincision length of two centimeters. In an example embodiment, duringinterval 13 of the surgical procedure, event E3 may be substantiallydifferent from event S3 (e.g., event E3 may be identified as an“incision” and event S3 may be identified as “suturing”). Duringinterval 14, event E4 may be substantially different from event S4 butmay be substantially the same (as indicated by arrow 2711, as shown inFIG. 27 ) as event S5 identified during interval 15. When calculatingthe deviation of sequence 2702 from 2701, event S4 of sequence 2702 maybe identified as an “inserted” event that does not have a correspondingcounterpart in sequence 2701. Such characterization of event S4 may berecorded (e.g., stored on a hard drive, database 111, or some otherlocation) for further analysis.

Aspects of disclosed embodiments may further include identifying anindication of a deviation between a specific surgical procedure and arecommended sequence of events for the surgical procedure. In somecases, identifying an indication of a deviation may include identifyingan indication of a deviation during an ongoing surgical procedure, suchas, for example, in real time during the surgical procedure. In variousembodiments, the deviation may be identified with a small delay asmeasured from the ongoing time of the surgical procedure due toprocessing related to identifying an indication of a deviation. Thedelay may be a millisecond, a second, a few seconds, a few tens ofseconds, a minute, a few minutes, and the like. Once the deviation isidentified, disclosure embodiments may include providing a notificationduring the ongoing surgical procedure. (e.g., provide the notificationas soon as the deviation is identified). For example, providing anotification may occur in real time during the surgical procedure.

Aspects of disclosed embodiments may include receiving an indicationthat a particular action is about to occur in a specific surgicalprocedure. The indication that the particular action is about to occurmay be based on an analysis of the frames of a surgical procedure. In anexemplary embodiment, the indication may be received from acomputer-based software application such as a machine-learning model foranalyzing surgical footage of an ongoing surgical procedure. Forexample, the machine-learning model may be an image recognitionalgorithm consistent with disclosed embodiments described herein.

In some embodiments, an image recognition algorithm may recognize asurgical tool in proximity to an anatomical structure and determine,based on the recognized surgical tool, that a particular action is aboutto occur in a surgical procedure. In some embodiments, the presence of asurgical tool, an anatomical structure, and/or an interaction between asurgical tool and an anatomical structure may serve as an indicator thata particular action is about to occur. As disclosed herein, an imagerecognition algorithm may analyze frames of a surgical procedure toidentify any of the forgoing. For example, the image recognitionalgorithm may determine a type of interaction between an instrument andan anatomical structure, a name of interaction, a name of an anatomicalstructure involved in the interaction, or any other identifiable aspectsof the interaction.

Additionally or alternatively, locations of healthcare professionals inan operating room, movements of any one of the healthcare professionals,hand motions of any one of the healthcare professionals, location and/orposition of a patient, placement of medical devices, and other spatialfeatures of healthcare professionals, patients, or instruments mayfurther indicate that a particular action is about to occur. In somecases, an indication that the particular action is about to occur may bebased on an input from a surgeon performing the specific surgicalprocedure. For example, audio sounds from any one of the healthcareprofessionals, gestures, or any other signals identifiable withinsurgical footage, audio data, image data, or device-based data (e.g.,data related to vital signs of a patient) may be used as an indicationthat a particular action is about to occur.

Disclosed embodiments may include identifying, using the recommendedsequence of events, a preliminary action to a particular action. Forexample, for a particular action such as suturing, a preliminary actionmay be clasping portions of an anatomical structure with forceps,administering a medication to a patient, repositioning image sensorswithin an operating room, measuring vital signals, connecting a medicaldevice to a patient (e.g., connecting an ECMO machine to a patient) orany other operation that needs to be performed prior to performing aparticular action.

Disclosed embodiments may further include determining, based on ananalysis of the accessed frames, that the identified preliminary actiondid not yet occur and in response, identifying the indication of thedeviation. In one example, determining that the identified preliminaryaction did not yet occur may be accomplished using image recognition, aspreviously discussed. For example, image recognition may identify thatpreliminary action did not yet occur by determining that a surgicalinstrument has not appeared in surgical footage or that there was nointeraction between a surgical instrument and an anatomical structure(as identified by analyzing surgical footage), or determining that thereare no changes to the anatomical structure (e.g., determining that ashape, color, size, or position of an anatomical structure isunchanged). Additionally or alternatively, image recognition maydetermine an absence of the preliminary action in other ways (e.g., bydetermining that healthcare professional has not yet approached apatient, by determining that an ECMO machine is not connected yet to apatient) or by using any other indication that may be identified insurgical footage. In an example embodiment, an indication of deviationbetween the specific surgical procedure and the recommended sequence ofevents may be the absence of the preliminary action. Alternatively, ifthe preliminary action is identified, one or more characteristics of thepreliminary action may be an indication of the deviation. For example,when preliminary action is an incision, the length of the incision maybe a characteristic of the preliminary action. If, for example, incisionlength is expected to be in a range of 10-20 cm, and the length isidentified to be 3 cm, such characteristic of the preliminary action mayindicate a deviation.

Aspects of disclosed embodiments may include providing a notification ofa deviation between the specific surgical procedure and the recommendedsequence of events before the particular action is performed. Thenotification may be any suitable electronic notification as describedherein and consistent with disclosed embodiments. Alternatively, thenotification may be any suitable sound signal, visual signal, or anyother signal (e.g., tactile signal, such as vibration) that may betransmitted to a healthcare professional (e.g., a surgeon administeringa surgical procedure).

Aspects of disclosed embodiments may include providing the notificationpostoperatively (i.e., after completion of the surgical procedure). Forexample, the deviation may be identified during or after the surgicalprocedure, and the notification may be provided after the deviation isevaluated using any one of (or any combination of) approaches describedabove. Additionally or alternatively, the deviation between the specificsurgical procedure and the recommended sequence of events for thesurgical procedure may be analyzed and/or evaluated by a healthcareprofessional.

Aspects of disclosed embodiments may include determining a name of anintraoperative surgical event associated with the deviation. Forexample, when a deviation between the specific surgical procedure andthe recommended sequence of events is identified, a name and/or a typeof event responsible for the deviation may be identified. For example, adeviation between an event of sequence 2702 and recommended sequence2701 is identified (e.g., when event E3 is substantially different fromevent S3), a name and/or type of event S3 (e.g., the name may be“suturing”) may be determined. Additionally, the name and/or type ofevent E3 may be determined. In an example embodiment, the name of eventS3 may be identified using a machine-learning image recognition model,as described above.

In various embodiments, a name of the intraoperative surgical eventassociated with the deviation may be the name of a preliminary actionprior to a particular action identified in a surgical event.Alternatively, a name of an intraoperative surgical event associatedwith the deviation may be the name of a particular action. In somecases, a name of an intraoperative surgical event may be a text stringcontaining multiple names of events or actions that contribute to thedeviation. In some cases, punctuation (or any other suitable means, suchas characters, paragraph marks, or new lines) may be used to separatedifferent names within the text string. For example, the name of anintraoperative surgical event associated with the deviation may be“clasping an artery with forceps; applying a laser beam; suturing theartery.”

In some embodiments, determining a name includes accessing a datastructure that correlates names with video footage characteristics. Adata structure may be any suitable data structure, such as structure1701, as shown in FIG. 17A. For example, determining a name may includeaccessing surgical footage (herein, also referred to as video footage)and determining video footage characteristics, such as events, actions,or event characteristics, as described in the present disclosure andconsistent with various embodiments of the disclosure.

In various embodiments, upon determining the name of the intraoperativesurgical event associated with a determined deviation, a notification ofthe deviation, including the name of the intraoperative surgical eventassociated with the deviation may be provided. In an example embodiment,the notification may be provided to various users (e.g., medicalpersonnel, administrators, and the like). In some cases, thenotification may be provided to patients, relatives or friends ofpatients, and the like. The notification may include text data, graphicsdata, or any other suitable data (e.g., video data, animations, or audiodata). Additionally or alternatively, the notification may beimplemented as a warning signal (e.g., light signal, audio signal, andthe like). In some cases, notification may be an SMS message, an email,and the like delivered to any suitable devices (e.g., smartphones,laptops, desktops, monitors, pagers, TVs, and the like) in possession ofvarious users authorized to receive the notification (e.g., variousmedical personnel, administrators, patients, relatives or friends ofpatients, and the like).

Aspects of disclosed embodiments may include receiving an inputindicating that a healthcare professional is about to perform an action.Such input may enable providing the notification of the deviation (forexample, of a skipped step required according to the recommendedsequence of events) before the action is taken by the surgeon. In somecases, such input from a surgeon or from another healthcare professionalmay include a press of a button, an audible input, a gesture, or anyother suitable input, as discussed above, indicating that the surgeon isabout to perform the particular action.

An action (about to be performed by a healthcare professional) may beany procedure-related action. For example, the action may includesuturing, incision, dissection, suctioning, placement of a cameraadjacent to or inside a body of a patient, or anything else that mayoccur during a procedure. In some cases, the action may includeadministering a medicine to a patient or measuring patient vital signalssuch as a pulse, a blood pressure, oxygen levels, and the like.

In various cases, receiving an input may include receiving an input fromthe healthcare professional. For instance, a surgeon may provide aninput via a visual or an audio signal (e.g., using a hand gesture, abody gesture, a visual signal produced by a light source generated by amedical instrument, a spoken word, and the like) that may be captured byone or more image sensors/audio sensors and recognized as an inputindicating that a healthcare professional is about to perform an action.In some cases, the healthcare professional may press a button, or useany other device (e.g., a smartphone, a laptop, and the like) to providethe input.

In some cases, the input may indicate what type of action is going to beperformed. For example, a surgeon may pronounce a name of the actionthat is about to be performed, and an audio signal from the surgeon maybe captured using a microphone. In an example embodiment, a speechrecognition model may be used to recognize one or more words pronouncedby the surgeon.

In some cases, receiving an input indicating that a healthcareprofessional is about to perform an action may include receiving theinput from a user who is not a healthcare professional. For example, theinput may be received from a person observing the surgical procedure.

Additionally or alternatively, the input may be received from amachine-learning algorithm that is trained to recognize various surgicalevents leading to possible future actions during surgical procedures.For example, the machine-learning algorithm may be configured torecognize that an incision is about to be performed based on a specificsurgical event, such as a surgeon holding and/or moving a scalpel in theproximity of an anatomical structure.

In various embodiments, an indication that the particular action isabout to occur may be an entrance of a particular medical instrument toa selected region of interest (ROI). For example, such indication may bedetermined using an object detection algorithm to detect the presence ofthe particular medical instrument in the selected ROI. In variousembodiments, a presence of a surgical tool in the proximity of a givenROI during a time (or time interval) of the surgical procedure may beused (for example, by a machine-learning model) to recognize that aparticular action is about to be taken. For different times during thesurgical procedure, the presence of the surgical tool in the proximityof the ROI may indicate different actions that are about to be taken. Insome cases, the method may include providing a notification when a givensurgical tool is present in the proximity of the ROI and forgoingproviding the notification when the surgical tool is not in the ROI. Asdescribed above, the notification may be any suitable notificationprovided to a healthcare professional, a healthcare administrator, oranyone else authorized to receive such information.

In various embodiments, identify that a particular medical instrumententered a selected region of interest (ROI) may be accomplished usingany suitable approach, such as using image recognition for analyzingframes of a surgical procedure, as described herein and consistent withdisclosed embodiments. In some cases, an ROI may be selected based on alocation of an anatomical structure. Or, if a second medical instrumentis used during a surgical procedure, an ROI may be selected based on alocation of a second medical instrument. Additionally or alternatively,an ROI may be selected based on a field of view of an image sensor. Forexample, a field of view of a particular image sensor (e.g., a sensorthat displays a magnified portion of an anatomical structure) may beused to select an ROI.

In various embodiments, based on the input indicating that a health careprofessional is about to perform an action, the method may includeaccessing the stored data structure identifying the recommended sequenceof events. The stored data structure may be any suitable data structuresuch as an array, an associative array, a linked list, a binary tree, abalanced tree, a heap, a stack, a queue, a set, a hash table, a record,a tagged union, an XML code, an XML database, an RDBMS database, an SQLdatabase, and the like. The data structure may include a recommendedsequence of events. For example, the data structure may list the namesof the events in a table with one event following the other.Alternatively, events may be organized and linked via a linked list. Invarious embodiments, the data structure may be any suitable datastructure that is configured to identify recommended events and to orderthe events to form a sequence.

Aspects of disclosed embodiments may further include detecting thepresence of a surgical tool in a predetermined anatomical region. Asused herein, the surgical tool may be any instrument or device that maybe used during a surgical procedure, which may include, but is notlimited to, cutting instruments (such as scalpels, scissors, saws,etc.), grasping and/or holding instruments (such as Billroth's clamps,hemostatic “mosquito” forceps, atraumatic hemostatic forceps, Deschamp'sneedle. Hopfner's hemostatic forceps, etc.), retractors (such asFarabefs C-shaped laminar hook, blunt-toothed hook, sharp-toothed hook,grooved probe, tamp forceps, etc.), tissue unifying instruments and/ormaterials (such as needle holders, surgical needles, staplers, clips,adhesive tapes, mesh, etc.), protective equipment (such as facial and/orrespiratory protective equipment, headwear, footwear, gloves, etc.),laparoscopes, endoscopes, patient monitoring devices, and so forth. Asurgical tool (also referred to as a medical tool or medical instrument)may include any apparatus or a piece of equipment used as part of amedical procedure.

An anatomical region may be any region that includes anatomicalstructures of a living organism. For example, the anatomical region mayinclude cavities (e.g., a surgical cavity), organs, tissues, ducts,arteries, cells, or any other anatomical parts. In some cases,prosthetics, artificial organs, and the like may be considered asanatomical structures and appear within anatomical regions. In oneexample, a machine learning model may be trained using training examplesto identify anatomical regions in images and/or videos, and the trainedmachine learning model may be used to analyze various captured frames ofthe surgical procedure and detect an anatomical region. An example ofsuch training example may include an image and/or a video, together witha label indicating an anatomical region within the image and/or withinthe video.

The presence of the surgical tool in a predetermined anatomical regionmay be detected using any suitable means. In an example embodiment, atrained machine learning model may be used to analyze various capturedframes of the surgical procedure to detect the presence of the surgicaltool in a predetermined anatomical region. The trained machine-learningmodel may be an image recognition model for recognizing an imagefeature, such as a surgical tool in a predetermined anatomical region.In various embodiments, based on the presence of the surgical tool in apredetermined anatomical region, the method may include accessing thestored data structure identifying the recommended sequence of events, asdiscussed above.

Aspects of preferred embodiments may further include identifying anindication of a deviation between the specific surgical procedure andthe recommended sequence of events for the surgical procedure bydetermining that a surgical tool is in a particular anatomical region.For example, if it is determined (e.g., using a machine-learning method,or using an indication from a healthcare professional) that the surgicaltool is present in a particular anatomical region, some embodiments maydetermine that a deviation has occurred. In some cases, if the surgicaltool is present in a particular anatomical region during a time (or atime interval) of the surgical procedure when it should not be present,some embodiments may determine that the deviation has occurred.Alternatively, in some cases, identifying an indication of a deviationmay include determining that a surgical tool is not in a particularanatomical region. For example, if during a time (or a time interval) ofthe surgical procedure, the surgical tool is not present in a particularanatomical region, some embodiments may be configured to determine thatthe deviation has occurred.

Additionally or alternatively, identifying an indication of a deviationmay include identifying an interaction between a surgical tool and ananatomical structure. A process of identifying the interaction between asurgical tool and an anatomical structure may involve analyzing framesof the surgical procedure to identify the interaction, for example asdescribed above. For example, at least some of the frames of thesurgical procedure may indicate a portion of the surgical procedure inwhich a surgical operation is being performed on the anatomicalstructure. As discussed above, the interaction may include any action bythe surgical tool that may influence the anatomical structure or viceversa. For example, the interaction may include a contact between thesurgical tool and the anatomical structure, an action by the surgicaltool on the anatomical structure (such as cutting, clamping, grasping,applying pressure, scraping, etc.), a physiological response by theanatomical structure, the surgical tool emitting light towards theanatomical structure (e.g., surgical tool may be a laser that emitslight towards the anatomical structure), a sound emitted towardsanatomical structure, an electromagnetic field created in a proximity ofthe anatomical structure, a current induced into an anatomicalstructure, or any other recognizable forms of interaction.

In some cases, identifying interaction may include identifying theproximity of the surgical tool to an anatomical structure. For example,by analyzing the surgical video footage of a surgical procedure, theimage recognition model may be configured to determine a distancebetween the surgical tool and a point (or a set of points) on a surfaceof an anatomical structure or within an anatomical structure.

In various embodiments, if the interaction between a surgical tool andan anatomical structure during a surgical procedure is identified and nosuch interaction is expected for a reference surgical procedure (i.e.,the surgical procedure that follows a recommended sequence of events),then an embodiment may be configured to determine that the deviation hasoccurred. Alternatively, if the interaction between a surgical tool andan anatomical structure is not identified (e.g., if the interaction isnot present during a surgical procedure), and the interaction isexpected for a reference surgical procedure, then an embodiment may beconfigured to determine that the deviation has occurred. Someembodiments may be configured to determine that there is no substantialdeviation of a surgical procedure and a reference surgical procedure ifan interaction between a surgical tool and an anatomical structure ispresent (or absent) in both the surgical procedure and the referencesurgical procedure.

Aspects of embodiments for enabling determination and notification of anomitted event in a surgical procedure are illustrated in FIG. 28 by aprocess 2801. At step 2811, process 2801 may include accessing frames ofvideo captured during a specific surgical procedure using any suitablemeans. For example, accessing may include accessing via a wired orwireless network via input devices (e.g., keyboard, mouse, etc.) or viaany other means for allowing reading/writing data.

At step 2813, process 2801 may include accessing stored data identifyinga recommended sequence of events for the surgical procedure, asdescribed above. At step 2815, process 2801 may include comparing theaccessed frames with the recommended sequence of events to identify anindication of a deviation between the specific surgical procedure andthe recommended sequence of events for the surgical procedure. Thedeviation between the specific surgical procedure and the recommendedsequence of events for the surgical procedure may be determined usingany suitable approaches described above (e.g., by calculating thedifference between different events using a suitable measure function,by using a machine-learning model, and so forth). At step 2817, process2801 may include determining a name of an intraoperative surgical eventassociated with the deviation using any suitable approach describedabove (e.g., using a machine-learning model to identify theintraoperative surgical event). Process 2801 may conclude with step 2819for providing a notification of the deviation, including the name of theintraoperative surgical event associated with the deviation. Asdescribed above, the notification may be any suitable notification(e.g., SMS text, video, images, etc.) and may be delivered to healthcareprofessionals, administrators, or any other authorized individual.

As previously discussed, the present disclosure relates to methods andsystems for enabling determination and notification of an omitted eventin a surgical procedure, as well as non-transitory computer-readablemedia that may include instructions that, when executed by at least oneprocessor, cause the at least one processor to execute operationsenabling determination and notification of an omitted event in asurgical procedure. The operations may include various steps of methodsfor enabling determination and notification of an omitted event in asurgical procedure, as described above.

Disclosed systems and methods may involve analyzing current and/orhistorical surgical footage to identify features of surgery, patientconditions, and other features to predict and improve surgical outcomes.Conventional approaches for providing decision support for surgicalprocedures may be unable to be performed in real time or may be unableto determine decision making junctions in surgical videos and developrecommendations to perform specific actions that improve surgicaloutcomes. In such situations, surgeons may miss critical decision makingpoints and/or fail to perform particular actions that can improveoutcomes, and surgeries may result in suboptimal outcomes for patients.In contrast, some embodiments of the present disclosure provideunconventional approaches that efficiently, effectively, and in realtime provide decision support for surgical procedures.

In accordance with the present disclosure, a method for providingdecision support for surgical procedures is disclosed. A surgicalprocedure may include a procedure performed by one or more surgeons. Asurgeon may include any person performing a surgical procedure,including a doctor or other medical professional, any person assisting asurgical procedure, and/or a surgical robot. A patient may include anyperson undergoing a surgical procedure. Non-limiting examples ofsurgical procedures may include inserting an implant into a patient,cutting, stitching, removing tissue, grafting, cauterizing, removing anorgan, inserting an organ, removing a limb or other body part, adding aprosthetic, removing a tumor, performing a biopsy, performing adebridement, a bypass, and/or any other action to treat or diagnose apatient. An implant or implant unit may include a stent, a monitoringunit, and/or any other material used within the body to replace amissing biological structure, support a damaged biological structure, orenhance an existing biological structure. Surgical tools, such aslaparoscopes, cameras, cutters, needles, drills, and/or any other deviceor implant may be used during a surgical procedure. In addition, duringa surgical procedure, medicine (such as an anesthetic drug, anintravenous fluid, a treatment drug, and/or any other compound orpreparation) may be used.

Decision support may include providing recommendations that may guidesurgeons in making decisions. Decision support may include analyzingvideo footage of prior similar surgical procedures, identifying a courseof action most likely to result in a positive outcome, and providing acorresponding recommendation to an operating surgeon. More generally,decision support for surgical procedures may include providinginformation to a medical professional during a surgical procedure, suchas a recommendation (in information illuminating a decision) to take oravoid an action. In some embodiments, decision support may includeproviding a computerized interface for alerting a medical professionalto a situation. An interface may include, for example, a display, aspeaker, a light, a haptic feedback component, and/or any other inputand/or feedback mechanism. In some embodiments, providing decisionsupport for surgical procedures may include providing real-timerecommendations to a surgeon (i.e., a method for providing decisionsupport for surgical procedures may be performed in real time during asurgical procedure). Real-time recommendations may include providingrecommendations via an interface in an operating room (e.g., anoperating room depicted in FIG. 1 ). Real-time recommendations may beupdated during a surgical procedure.

In some embodiments, a method may include receiving video footage of asurgical procedure performed by a surgeon on a patient in an operatingroom. Video footage may include video captured by one or more camerasand/or sensors. Video footage may include continuous video, video clips,video frames, an intracavitary video, and/or any other video footage.Video footage may depict any aspect of a surgical procedure and maydepict a patient (internally or externally), a medical professional, arobot, a medical tool, an action, and/or any other aspect of a surgicalprocedure. In some embodiments, video footage may include images from atleast one of an endoscope or an intracorporeal camera (e.g., images ofan intracavitary video). An endoscope may include a rigid or flexibletube, a light, an optical fiber, a lens, an eyepiece, a camera, acommunication component (e.g., a wired or wireless connection), and/orany other component to assist in collecting and transmitting images fromwithin a patient's body. An intracorporeal camera may include any imagesensor used to collect images from within a patient's body before,during, or after a surgical procedure.

Receiving video footage may occur via a sensor (e.g., an image sensorabove a patient, within a patient, or located elsewhere within anoperating room), a surgical robot, a camera, a mobile device, anexternal device using a communication device, a shared memory, and/orany other connected hardware and/or software component capable ofcapturing and/or transmitting images. Video footage may be received viaa network and/or directly from a device via a wired and/or wirelessconnection. Receiving video footage may include reading, retrieving,and/or otherwise accessing video footage from data storage, such as adatabase, a disk, a memory, a remote system, an online data storage,and/or any location or medium where information may be retained.

Consistent with disclosed embodiments, an operating room may include anyroom configured for performing surgery, including a room in a hospital,in a clinic, in a temporary clinic (e.g., a room or tent configured forsurgery during a disaster relief or war event), and/or any in any otherlocation where surgical procedures may be performed. An exemplaryoperating room is depicted in FIG. 1 .

Consistent with disclosed embodiments, a method for providing decisionsupport for surgical procedures may include accessing at least one datastructure including image-related data characterizing surgicalprocedures. Accessing a data structure may include receiving data of adata structure via a network and/or directly from a device via a wiredand/or wireless connection. Accessing a data structure may includeretrieving data of a data structure from data storage, consistent withsome disclosed embodiments.

Consistent with the present embodiments, a data structure may includeprimitive types, such Boolean, character, floating point, integer,reference, and enumerated type; composite types such as container, list,tuple, multimap, associative array, set, multiset, stack, queue, graph,tree, heap; any form of hash-based structure or graph. Further examplesmay include relational databases, tabular data, and/or other form ofinformation organized for retrieval. Data within the data structure maybe organized following a data schema including a data type, a key-valuepair, a label, metadata, a field, a tag, an index, and/or other indexingfeature.

Video and/or image-related data characterizing surgical procedures maybe included within the data structure. Such image-related data mayinclude video-characterizing information and/or some or all of the videofootage itself, images, and/or a preprocessed version of the videoand/or image data. In another example, such video and/or image-relateddata may include information based on an analysis of the video and/orimage. In yet another example, such video and/or image-related data mayinclude information and/or one or more rules for analyzing image data.One example of a data structure is illustrated in FIG. 17A.

Consistent with disclosed embodiments, image-related data characterizingsurgical procedures may include data relating to an eventcharacteristic, an event location, an outcome, a deviation between asurgical procure and a mandatory sequence of events, a skill level, anevent location, an intraoperative surgical event, an intraoperativesurgical event characteristics, a characteristic event, a leakagesituation, an event within a surgical phase, a tag, a mandatory sequenceof events, an omitted event, a recommended sequence of event, ananatomical structure, a condition, contact between an anatomicalstructure and a medical instrument, an interaction, and/or any otherinformation describing or defining aspects of surgical procedures.

In some embodiments, a method for providing decision support forsurgical procedures may include analyzing received video footage usingimage-related data to determine an existence of a surgical decisionmaking junction. A surgical decision making junction may include a time(e.g., a time-point or time period) in a surgical video. For example, itmay relate to an event or situation that poses an opportunity to pursuealternative courses of action. For example, a decision making junctionmay reflect a time in which a surgeon may take one or more actions tochange a surgical outcome, to follow a surgical procedure, to change toa different surgical procedure, to deviate from a surgical procedure,and/or to vary any other approach.

Analyzing received video footage may include performing methods of imageanalysis on one or more frames of received video footage, consistentwith disclosed embodiments. Analyzing received video footage mayinclude, for example, methods of object recognition, imageclassification, homography, pose estimation, motion detection, and/orother video analysis methods, for example as described above. Analyzingreceived video footage may include using a trained machine learningmodel, and/or training and/or implementing a machine learning model,consistent with disclosed embodiments. For example, received videofootage may be analyzed using a machine learning model trained usingtraining examples to detect and/or identify a surgical decision juncturefrom images and/or videos. For example, received video footage may beanalyzed using an artificial neural network configured to detect and/oridentify a surgical decision juncture from images and/or videos. In someembodiments, received video footage may be compared with image-relateddata to determine an existence of a surgical decision juncture. This mayoccur, for example, through video analysis, and may occur in real time.(E.g., as video is captured of the surgeon operating, analysis may beperformed on the video in real time, and surgical junctions may beidentified.) In one example, the image-related data may comprise one ormore rules for analyzing image data (such as trained machine learningmodels, artificial neural networks, etc.), and the one or more rules maybe used to analyze the received video footage to determine the existenceof the surgical decision making junction. In one example, a Markov modelmay be utilized based on an analysis of frames from the received videofootage to determine the existence of the surgical decision makingjunction. In other examples, an artificial neural network (such as aRecurrent Neural Network or a Long Short-Term Memory neural network) maybe used to analyze the received video footage and determine theexistence of the surgical decision making junction.

By way of example, a decision making junction may arise upon detectionof an inappropriate access or exposure, a retraction of an anatomicalstructure, a misinterpretation of an anatomical structure or a fluidleak, and/or any other surgical event posing an opportunity to pursuealternative courses of action. Inappropriate access or exposure mayinclude opening and/or cutting a wrong tissue, organ, and/or otheranatomical feature. Retraction may involve movement, traction, and/orcounter-traction of tissues to expose tissue, or organ, and/or otheranatomical structure for viewing by a surgeon. A misinterpretation of ananatomical structure or fluid leak may include a misclassification(e.g., classification of a wrong structure or fluid type) and/or anincorrect estimation of a source and/or severity of a fluid leak. Moregenerally, misinterpretation may include any incorrect conclusionreached by a system or person during a surgical procedure.

In some embodiments, a decision making junction may be determined by ananalysis of a plurality of differing historical procedures wherediffering courses of action occurred following a common surgicalsituation. For example, a plurality of differing historical proceduresmay be included in historical video footage and/or in received videofootage. Historical procedures may depict one or more surgicalprocedures, one or more patients, one or more conditions, one or moreoutcomes, and/or one or more surgeons. In some embodiments, differingcourses of action may include differing actions during surgicalprocedures, as described herein. Differing courses of action may includeactions which are not the same (e.g., an action to suture a lacerationand an action to staple a laceration may be considered differingactions). Differing courses of action may include different methods ofperforming a same action (e.g., applying one contact force and applyinganother contact force may be different methods of performing a sameaction). Differing courses of action may include using different medicaltools. A common surgical situation may refer to a situation thatincludes a type of surgical procedure (such as a cholecystectomy), asurgical event (e.g., an incision, a fluid leakage event, etc.), and/orany other aspect of a surgery that may be common to a plurality ofhistorical surgical procedures.

In some embodiments, determining a presence of a decision makingjunction may be based on a detected physiological response of ananatomical structure and/or a motion associated with a surgical tool. Aphysiological response may include a movement of an anatomicalstructure, a leakage, and/or any other physiological activity. Aphysiological response may include a change in a heart rate, a breathingrate, a blood pressure, a temperature, a blood flow, and/or a change inany other biological parameter or health status. Other non-limitingexamples of possible physiological responses are described above. Amotion associated with a surgical tool any include any movement (e.g.,translation and/or rotation) of a surgical tool. A surgical tool mayinclude any surgical tool, as disclosed herein. Detecting aphysiological response and/or a motion associated with a surgical toolmay include performing a method of image analysis, as also describedherein.

In some embodiments, a method for providing decision support forsurgical procedures may include accessing, in at least one datastructure, a correlation between an outcome and a specific action takenat a decision making junction. Accessing a correlation may includedetermining an existence of a correlation, reading a correlation frommemory, and/or determining in any other manner that a correlation existsbetween a particular action and an outcome. In some embodiments, acorrelation may be accessed in a data structure based on an index, theindex including at least one of a tag, a label, a name, or otheridentifier of a specific action, a decision making junction, and/or anoutcome. In some embodiments, accessing a correlation may includedetermining (e.g., generating, looking up, or identifying) a correlationusing an algorithm such as a model, a formula, and/or any other logicalapproach. Consistent with disclosed embodiments, a correlation mayindicate a probability (e.g., likelihood) of a desired outcome (e.g.,positive outcome) and/or undesired outcome (e.g., negative outcome)associated with a specific action. A correlation may include acorrelation coefficient, a goodness of fit measure, a regressioncoefficient, an odds ratio, a probability, and/or any other statisticalor logical interrelationship. In one example, one correlation may beused for all decision making junction of a particular type, while inanother example, a plurality of correlations may be used for differentsubsets of the group of all decision making junction of the particulartype. For example, such subset may correspond to a particular group ofpatients, to a particular group of surgeons (and/or other healthcareprofessionals), to a particular group of surgeries, to a particulargroup of operating rooms, to particular previous events in the surgicalprocedure, to any union or intersection of such groups, and so forth.

A specific action may include any action performed by a surgeon (e.g., ahuman or robotic surgeon) during a surgical procedure, or by a person orrobot assisting a surgical procedure. Examples of specific actions mayinclude remedial actions, diagnostic actions, actions following asurgical procedure, actions deviating from a surgical procedure, and/orany other activity that might occur during a surgical procedure. Suchactions may include engaging a medical instrument with a biologicalstructure, administering a medication, cutting, suturing, alteringsurgical contact, conducting a medical test, cleaning an anatomicalstructure, removing excess fluid, and/or any other action that may occurduring a surgical procedure.

A specific action may include a single step or a plurality of steps(e.g., a plurality of actions performed during a surgery). A step mayinclude any action or subset of an action as described herein.Non-limiting examples of specific actions may include one or more ofsteps to make an incision, to insert an implant, to attach an implant,and to seal an incision.

In some embodiments, a specific action may include introducing anadditional surgeon to an operating room. For example, the additionalsurgeon may have more experience, a higher skill level, a particularexpertise (e.g., a technical expertise, a particular problem-solvingexpertise, and/or other expertise), than a surgeon already present inthe operating room. Bringing a surgeon to an operating room may includetransmitting a notification requesting or instructing a surgeon to cometo an operating room. In some embodiments, an additional surgeon may bea surgical robot, and bringing an additional surgeon to an operatingroom may include activating the robot and/or providing instructions tothe robot to perform and/or assist a surgical procedure. Providinginstructions to a robot may include instructions to perform one or moreactions.

In some embodiments, a method for providing decision support forsurgical procedures may include outputting a recommendation to a user toundertake and/or to avoid a specific action. Such a recommendation mayinclude any guidance, regardless of the form of the guidance (e.g.,audio, video, text-based, control commands to a surgical robot, or otherdata transmission that provides advice and/or direction). In someinstances, the guidance may be in the form of an instruction, in othersit may be in the form of a recommendation. The trigger for such guidancemay be a determined existence of a decision-making junction and anaccessed correlation. Outputting a recommendation may includetransmitting a recommendation to a device, displaying a recommendationon an interface, and/or any other mechanism for supplying information toa decision maker. Outputting a recommendation to a user may includeoutputting a recommendation to a person in an operating room, to asurgeon (e.g., a human surgeon and/or a surgical robot), to a personassisting a surgical procedure (e.g., a nurse), and/or any to otheruser. For example, outputting a recommendation may include transmittinga recommendation to a computer, a mobile device, an external device,smart glasses, a projector, a surgical robot, and/or any other devicecapable of conveying information to the user. In some embodiments, asurgeon may be a surgical robot and a recommendation may be provided inthe form of an instruction to the surgical robot (e.g., an instructionto undertake a specific action and/or avoid a specific action).

Outputting a recommendation may occur via a network and/or via a directconnection. In some embodiments, outputting a recommendation may includeproviding output at an interface in an operating room. For example,outputting a recommendation may include causing a recommendation to bepresented via an interface (e.g., a visual and/or audio interface in anoperating room). In some embodiments, outputting a recommendation mayinclude playing a sound, altering a light (e.g., turning a light on oroff, pulsing a light), providing a haptic feedback signal, and/or anyother method of alerting a person or providing information to a personor surgical robot.

By way of example, a recommendation may include a recommendation toconduct a medical test. In some embodiments, a medical test may includea blood analysis, a medical imaging of a patient, a urine analysis, datacollection by a sensor, and/or any other analysis. Medical imaging mayinclude an intraoperative medical imaging (i.e., an imaging that occursduring a surgical procedure), such as X-ray imaging, computerizedtomography (CT), medical resonance imaging (MRI), other proceduresinvolving a contrasting agent, ultrasound, or other techniques forcreating body part images for diagnostic and/or treatment purposes.

In some embodiments, a method for providing decision support forsurgical procedures may include outputting a recommendation (e.g., afirst recommendation, second recommendation, and/or an additionalrecommendation) to a user to undertake or to avoid a specific actionbased a determined existence of a decision making junction, an accessedcorrelation, and a received result of a medical test. A method forproviding decision support for surgical procedures may therefore includereceiving a result of a medical test. A result of a medical test mayinclude medical data, sensor data, instrument data, and/or any otherinformation reflective of a biological condition. A result of a medicaltest may include an indicator of a health status and/or a condition of apatient. A result may include, for example, a presence or absence of abiomarker, a presence or absence of a tumor, a location of an anatomicalfeature, an indicator of metabolic activity (e.g., glucose uptake), anenzyme level, a heart status (e.g., heart rate), a temperature, abreathing indicator, and/or any other health or condition indicator. Aresult may be received via network and/or from a connected device.Receiving a result may include receiving and/or accessing a datastorage, consistent with disclosed embodiments. For example, in responseto a first value of the received result of the medical test, arecommendation to undertake (or to avoid) a first action may beoutputted, and in response to a second value of the received result ofthe medical test, outputting the recommendation to undertake (or toavoid) the first action may be withheld.

In some embodiments, a recommendation may include a name and/or otheridentifier (e.g., an employee ID) of an additional surgeon. In someembodiments, outputting a recommendation may include providing anindication to an additional surgeon. An indication may include anotification, an alert, a request to come to an operating room, a resultof a medical test, information indication that assistance may be neededduring a surgical procedure, and/or any other indication. In oneexample, the additional surgeon may be selected (for example, from aplurality of alternative additional surgeons) based on one or more of acharacteristic of the patient undergoing the surgical procedure, thesurgeon currently performing the surgical procedure, the operating room,a tool used in the surgical procedure, a condition of an anatomicalstructure related to the surgical procedure, an interaction between amedical instrument and an anatomical structure in the surgicalprocedure, a physiological response related to the surgical procedure,characteristics of the additional surgeon, and so forth.

Consistent with the present embodiments, a recommendation may include adescription of a current surgical situation, guidance, an indication ofpreemptive or corrective measures, an indication of alternativeapproaches, danger zone mapping, and/or any other information that mightinform the surgeon relative to a surgical procedure. A description of acurrent surgical situation may include a health status and/or acondition of a patient (e.g., a condition reflected in sensor data suchas heart rate monitor data, brain activity data, temperature data,leakage data, and/or any other health data). A description of a currentsurgical situation may also or alternatively include an evaluation of acurrent or possible future outcome. A preemptive measure and/or acorrective measure may include an action to follow and/or change asurgical procedure. A preemptive measure and/or a corrective measure mayinclude any action by a surgeon and/or person assisting a surgery,and/or an action that may result in avoiding a negative outcome. Acorrective measure may include an action that may improve an outcome. Insome embodiments, danger zone mapping may include identifying one ormore specific actions and likely outcomes (e.g., a set of specificactions associated with negative outcomes such as death, disability, orother undesirable eventuality). Danger zone mapping may include, forexample, identification of anatomical regions that if not accessedproperly, may adversely impact patient safety and surgery outcomes. Forexample, in inguinal hernia, danger zones may include the ‘triangle ofdoom’ that lies between the Vas deferens in men or round ligament of theuterus in women (medially) and the testicular vessels in men(laterally), and holds important structures such as iliac vessels,femoral nerve, genital branch of the genitofemoral nerve, and/or the‘triangle of pain’ that lies between the testicular vessels (medially),the psoas muscle (laterally) and the ileopubic tract (superiorly) andholds important structures such as the femoral branch of thegenitofemoral nerve and the lateral femoral cutaneous nerve, arecritical. Injuries to structures within the “triangle of doom” may, insome cases, be fatal. Injuries to structures within the “triangle ofpain” may, in some cases, result in chronic pain. In some examples, amachine learning model may be trained using training examples toidentify danger zones in surgical images and/or surgical videos, and thetrained machine learning model may be used to analyze the video footageand identify and/or map the danger zones. An example of such trainingexample may include an image and/or a video, together with a labelindicating the danger zones depicted in the image and/or in the video.In one example, a description of a danger zone mapping may includetextual description of relevant identified danger zones. In anotherexample, a description of a danger zone mapping may include visualmarking of relevant identified danger zones, for example as an overlayover at least one frame of the video footage, in an augmented realitysystem, and so forth.

By way of example, a recommendation may include a recommended placementof a surgical drain, such as to drain inflammatory fluid, blood, bile,and/or other fluid from a patient.

A recommendation may include a confidence level that a desired surgicaloutcome will occur if a specific action is taken, and/or a confidencelevel that a desired outcome will not occur if a specific action is nottaken. A confidence level may be based on an analysis of historicalsurgical procedures, consistent with disclosed embodiments, and mayinclude a probability (i.e., likelihood) that an outcome will occur. Adesired outcome may be a positive outcome, such as an improved healthstatus, a successful placement of a medical implant, and/or any otherbeneficial eventuality. In some embodiments, a desired outcome mayinclude an avoidance of a possible undesired situation following adecision making junction (e.g., an avoidance of a side effect, apost-operative complication, a fluid leakage event, a negative change ina health status of a patient, and/or any other undesired situation).

In some embodiments, outputting a recommendation may be based on a timeelapsed since a particular point in a surgical procedure. For example, arecommendation may be based on a time elapsed since a surgical event,consistent with disclosed embodiments. A recommendation may be based ona surgical event that occurred at least a specified number of minutesbefore a decision making junction. In some embodiments, a surgical eventmay include a past action performed by a surgeon prior to a decisionmaking junction. A recommendation may also include an alternative courseof action. A course of action may include a set, a sequence, and/or apattern of actions. An alternative course of action may differ fromactions associated with an ongoing surgical procedure being followed bya surgeon.

In some embodiments, a recommendation may include an indication of anundesired surgical outcome likely to occur if a specific action is notundertaken. Such an indication may include a confidence level, adescription of an undesired surgical outcome (e.g., a name of anoutcome), and/or any other indication.

In some embodiments, a recommendation may be based on a skill level of asurgeon. For example, a surgeon with a high skill level may receive adifferent recommendation than a surgeon with a lower skill level. Insome embodiments, a recommendation may include a specific actionselected from a plurality of alternative actions, and a selection of aspecific action may be based on a skill level of a surgeon andcomplexity levels associated with a plurality of alternative actions. Askill level may be based on a historical performance score, a number ofsurgeries performed, overall time spent as a surgeon (e.g., a number ofyears; number of hours spent in surgery), an indication of a level oftraining, a classification of a surgeon's skill, and/or any otherassessment of a surgeon's skill whether derived from manual input, dataanalysis, or video image analysis.

In some embodiments, a recommendation may be based on a surgical eventthat occurred in a surgical procedure prior to a decision makingjunction (i.e., a prior surgical event). A prior surgical event mayinclude any surgical event as described herein, and which preceded thedecision making junction. A prior surgical event may be correlated witha positive or negative outcome after a decision making junction, and arecommendation may include a recommendation to perform a specific actionthat increases the likelihood of achieving a later positive outcome orof avoiding a later negative outcome. Thus, such a method may includedetermining that a prior surgical event is correlated with a lateroutcome. Such a correlation may be time-based, in that the correlationmay be determined based on an elapsed time between a surgical event andthe decision making junction.

In some embodiments, outputting a recommendation may include presentinga first instruction to perform a first step, receiving an indication ofthat a first step was performed successfully, and, in response to thereceived indication that a first step was performed successfully,presenting a second instruction to perform a second step. In someembodiments, outputting a recommendation may include presenting a firstinstruction to perform a first step and receiving an indication that thefirst step was not performed successfully. In some embodiments,outputting a recommendation may include forgoing presenting a secondinstruction in response to a received indication that a first step wasnot performed successfully. In some embodiments, in response to areceived indication that a first step was not performed successfully,outputting a recommendation may include presenting an alternativeinstruction to perform an alternative step, the alternative stepdiffering from a second step.

An indication that a first step was performed successfully orunsuccessfully may be based on an analysis of video footage, consistentwith disclosed embodiments. Receiving an indication may includereceiving video footage after presenting an instruction to perform afirst step and generating an indication based on an analysis of videofootage.

In some embodiments, a method for providing decision support forsurgical procedures may include receiving a vital sign of a patient, anda recommendation may be based on an accessed correlation and a vitalsign. A vital sign may be received from a medical instrument, a device,an external device, a data storage, a sensor, and/or any other computingcomponent, and may include any indicator a condition of a patient healthstatus (e.g., a heart rate, a breathing rate, a brain activity, and/orother vital sign). In some embodiments, vital signs may be received viaa network from a connected device, and may be detected either via atraditional sensor or through analysis of video footage.

In some embodiments, a recommendation may be based on a condition of atissue of a patient and/or a condition of an organ of a patient.Generally, a condition of a tissue or an organ may refer to anyinformation that indicates to a state or characteristic of a tissue ororgan. For example, a condition may be based on an assessment such aswhether tissue or organ is normal, abnormal, damaged, leaking, hydrated,oxygenated, dehydrated, retracted, enlarged, shrunken, present, absent,and/or any other appearance or status. Consistent with disclosedembodiments, a condition of a tissue and/or organ of a patient may bedetermined based on an analysis of video footage. For example, such ananalysis may determine a color of a tissue, a texture of an anatomicalstructure, a heart rate, a lung capacity, a presence of a lump or otherirregularity and/or any other characteristic of an anatomical structure.In some embodiments, a recommendation may be based on a conditionreflected in sensor data such as heart rate monitor data, brain activitydata, temperature data, leakage data, and/or any other health data.

As another example, a recommendation of a specific action may include asuggestion or direction to form a stoma, or a particular type of a stoma(e.g., loop stoma, end stoma, loop colostomy, end colostomy, loopileostomy, end ileostomy, urostomy, and/or any other type of stoma). Therecommendation may suggest a stoma creation technique, an indication ofa portion of a colon and/or ileum for creation of a stoma, and/or alocation on a skin of a patient for creation of a stoma. Or, arecommendation may suggest that a stoma not be created when, forexample, a creation of a stoma is correlated to an undesirable outcome.

A recommendation to create or avoid creating a stoma (or to take anyother course of action) may be based on a physiological impact on apatient, and a threshold of a measure of a possible improvement to anoutcome. A threshold may be selected based on a patient characteristic(e.g., an age, a prior health status, a family history, a vital sign,and/or other characteristic). For example, a lower threshold may beselected for a patient who previously had a stoma associated with adesired outcome. A threshold may also be based on whether a patient wasinformed of a possibility of a stoma prior to a surgery.

One example of a decision making junction may include deciding whetheror not to mobilize the ileum and/or the cecum, for example in thepreparation phase of an appendectomy, and the recommendation may includea suggestion to mobilize the ileum and/or the cecurn or a suggestion notto mobilize the ileum and/or the cecum. Some non-limiting examples offactors that may influence the decision may include procedure complexitylevel, age of the patient, the gender of the patient, previousinflammation and prior surgery. The recommendation may be based on atleast one of these factors. The decision made at this junction mayimpact the ability to resect the diseased appendix. Another example of adecision making junction may include deciding if the appendix can besafely divided or not, for example in the dissection and skeletonizationphase of an appendectomy, and the recommendation may include asuggestion to dissect or not to dissect the appendix. Some non-limitingexamples of factors that may influence the decision may includeprocedure complexity level, achieving a free appendix, and whether ornot ileum/cecum was mobilized properly. The recommendation may be basedon at least one of these factors. The decision made at this junction maydictate whether or not there will be the recurrence of appendicitis(‘stump appendicitis’). Another example of a decision making junctionmay include deciding what instrument to use for the division of theappendix, for example in the division phase of appendectomy, and therecommendation may include a suggestion of an instrument for thedivision. Some non-limiting examples of factors that may influence thedecision may include procedure complexity level, whether or not acircular view of the appendix was achieved, and patient body mass index.The recommendation may be based on at least one of these factors. Thedecision made at this junction may influence the length and cost oftreatment. Another example of a decision making junction may includedeciding whether or not to treat an appendiceal stump, for example inthe division phase of an appendectomy. Some options that may includeavoiding action for treating the appendiceal stump, to cauterize, or tooversew. A recommendation may include a suggestion of whether to treatthe appendiceal stump, and/or a suggestion of a particular action to betaken for treating the appendiceal stump. Some non-limiting examples offactors that may influence the decision may include procedure complexitylevel and which instrument was used to divide the appendix. Therecommendation may be based on at least one of these factors. Thedecision made at this junction may influence postoperative infection andfistulae rates. Another example of a decision making junction mayinclude deciding how to remove the resected sample (e.g., in an endobagor through the trocar), for example in the packaging phase ofappendectomy, and the recommendation may include a suggestion on how toremove a resected sample. For example, the decision may be based on theprocedure complexity level. The decision made at this junction mayinfluence surgical site infection rate Another example of a decisionmaking junction may include deciding whether or not to performirrigation, for example in the final inspection phase of appendectomy,and the recommendation may include a suggestion to perform irrigation ora suggestion not to perform irrigation. Some non-limiting examples offactors that may influence the decision may include procedure complexitylevel, patient pre-existing comorbidities, and patient gender. Therecommendation may be based on at least one of these factors. Thedecision made at this junction may influence infection rate. Anotherexample of a decision making junction may include deciding whether ornot to place a drain, for example in the final inspection phase ofappendectomy, and the recommendation may include a suggestion to place adrain or a suggestion not to place a drain. Some non-limiting examplesof factors that may influence the decision may include procedurecomplexity level, patient age, and patient pre-existing comorbidities.The recommendation may be based on at least one of these factors. Thedecision made at this junction may influence infection rate,complication rate and postoperative length of stay.

One example of a decision making junction in an access phase of alaparoscopic cholecystectomy may include a selection of an insertionmethod (such as Veres needle, Hasson technique, OptiView) and/or aselection of port placement arrangement (such as ‘Regular’ and‘Alternative’), and the recommendation may include a suggestion of aninsertion method and/or a suggestion of a port placement arrangement.One example of a decision making junction in an adhesiolysis phase of alaparoscopic cholecystectomy may include a selection of whether todecompress the gallbladder, and the recommendation may include asuggestion of whether to decompress the gallbladder. For example, whenthe gallbladder is distended and/or tense, or when other signs of acutecholecystitis are present, the recommendation may include a suggestionto decompress the gallbladder. One example of a decision making junctionin a laparoscopic cholecystectomy may include a selection of agallbladder dissection approach (such as Traditional, Dome-downDissection, Sub-total, and so forth), and the recommendation may includea suggestion of a gallbladder dissection approach. For example, in caseof a severe cholecystitis, a recommendation of a Dome-down Dissectionmay be provided. In another example, in case of an inability to obtainexposure, a recommendation to bail out may be provided, for example dueto an increase risk for large collaterals in the liver bed. One exampleof a decision making junction in a laparoscopic cholecystectomy mayinclude a selection of whether or not to place a drain, and therecommendation may include a suggestion to place a drain or a suggestionnot to place a drain.

In some examples, the recommendation to the user to undertake and/or toavoid the specific action to be outputted may be determined using atrained machine learning model. For example, a machine learning modelmay be trained using training examples to determine recommendationsbased on information related to surgical decision making junctions, andthe trained machine learning model may be used to determine therecommendation to be outputted to the user to undertake and/or to avoidthe specific action for a particular occurrence of a surgical decisionmaking junction based on information related to the particularoccurrence of the surgical decision making junction. Some non-limitingexamples of such information related to an occurrence of a surgicaldecision making junction are described above. For example, theinformation may include a type of the surgical decision making junction,properties of the surgical decision making junction, time of thesurgical decision making junction (e.g., within the surgical procedure),characteristics of a patient undergoing the surgical procedure,characteristics of a surgeon (or another healthcare professional)performing at least part of the surgical procedure, characteristics ofan operating room related to the surgical procedure, an anatomicalstructure related to the surgical decision making junction, a conditionof the anatomical structure related to the surgical decision makingjunction, a medical instrument used in the surgical procedure, aninteraction between a medical instrument and an anatomical structure inthe surgical procedure, a physiological response related to the surgicaldecision making junction, one or more surgical events that occurred inthe surgical procedure prior to the surgical decision making junction,duration of the one or more surgical events that occurred in thesurgical procedure prior to the surgical decision making junction,duration of surgical phases in the surgical procedure, one or morecorrelations between outcomes and possible actions that may be taken atthe surgical decision making junction, past responses of the user topreviously provided recommendations, and so forth. An example of suchtraining example may include information related to a surgical decisionmaking junction, together with a label indicating a desiredrecommendation. For example, the label may include a desired textualand/or graphical content for the desired recommendation. In anotherexample, the label may be based on a correlation between an outcome anda specific action taken at such surgical decision making junction.

FIG. 29 is a flowchart illustrating an example process 2900 for decisionsupport for surgical procedures, consistent with disclosed embodiments.Process 2900 may be performed using at least one processor, such as oneor more microprocessors. In some embodiments, process 2900 is notnecessarily limited to steps illustrated, and any of the variousembodiments described herein may also be included in process 2900. Asone of skill in the art will appreciate, steps of process 2900 may beperformed by a system including, for example, components of system 1401.In some embodiments, a non-transitory computer readable medium includinginstructions that, when executed by at least one processor, cause the atleast one processor to execute operations for providing decision supportfor surgical procedures according to process 2900. In some embodiments,process 2900 may be performed in real time during a surgical procedure.Based on the steps described in process 2900, the surgeon or other usersmay be able to more effectively and more efficiently perform surgicalprocedures with positive outcomes and/or avoid negative outcomes.

At step 2902, the process may include receiving video footage of asurgical procedure performed by a surgeon on a patient in an operatingroom, consistent with disclosed embodiments and as previously describedby way of examples. FIG. 1 provides an example of an operating room,surgeon, patient, and cameras configured for capturing video footage ofa surgical procedure. Video footage may include images from at least oneof an endoscope or an intracorporeal camera (e.g., images of anintracavitary video).

At step 2904, the process may include accessing at least one datastructure including image-related data characterizing surgicalprocedures, consistent with disclosed embodiments and as previouslydescribed by way of examples. In some embodiments, accessing a datastructure may include receiving data of a data structure via a networkand/or from a device via a connection. Accessing a data structure mayinclude retrieving data from a data storage, consistent with disclosedembodiments.

At step 2906, the process may include analyzing the received videofootage using the image-related data to determine an existence of asurgical decision making junction, consistent with disclosed embodimentsand as previously describe by way of examples. Analyzing received videofootage may include performing methods of image analysis on one or moreframes of received video footage, consistent with disclosed embodiments.Analyzing received video footage may include implementing a modeltrained to determine an existence of a surgical decision makingjunction. A decision making junction may include an inappropriate accessor exposure, a retraction of an anatomical structure, amisinterpretation of an anatomical structure or a fluid leak, and/or anyother surgical event, as previously described. In some embodiments, adecision making junction may be determined by an analysis of a pluralityof differing historical procedures where differing courses of actionoccurred following a common surgical situation. In some embodiments,determining a presence of a decision making junction may be based on adetected physiological response of an anatomical structure and/or amotion associated with a surgical tool.

At step 2908, the process may include accessing, in the at least onedata structure, a correlation between an outcome and a specific actiontaken at the decision making junction, as previously described by way ofexamples. As discussed, a specific action may be correlated with apositive or negative outcome, consistent with disclosed embodiments.Accessing a correlation may include generating a correlation, reading acorrelation from memory and/or any other method of accessing acorrelation in a data structure. A specific action may include a singlestep or a plurality of steps (e.g., a plurality of actions performed bya surgeon). A specific action may include summoning an additionalsurgeon to the operating room.

At step 2910, the process may include outputting a recommendation to auser to undertake the specific action, consistent with disclosedembodiments, as previously described by way of examples. Outputting arecommendation may be based on a determined existence of a decisionmaking junction and an accessed correlation, consistent with the presentembodiments. In some embodiments, outputting a recommendation mayinclude providing output via an interface in an operating room. In someembodiments, a surgeon is a surgical robot and a recommendation may beprovided in the form of an instruction to the surgical robot (e.g., aninstruction to undertake a specific action and/or avoid a specificaction). By way of example, a recommendation may include arecommendation to conduct a medical test. A recommendation (e.g., afirst recommendation, second recommendation, and/or an additionalrecommendation) may include a recommendation to the user to undertake orto avoid a specific action based a determined existence of a decisionmaking junction, an accessed correlation, and a received result of amedical test. A recommendation may include a name and/or otheridentifier (e.g., an employee ID) of an additional surgeon. Arecommendation may include a description of a current surgicalsituation, an indication of preemptive or corrective measures, and/ordanger zone mapping. In one example, as previously mentioned, arecommendation may include a recommended placement of a surgical drainto remove inflammatory fluid, blood, bile, and/or other fluid from apatient. A confidence level that a desired surgical outcome will or willnot occur if a specific action is taken or not taken may be part of arecommendation. A recommendation may be based on a skill level of asurgeon, a correlation and a vital sign, and/or a surgical event thatoccurred in a surgical procedure prior to a decision making junction(i.e., a prior surgical event). In some embodiments, a recommendationmay be based on a condition of a tissue of a patient and/or a conditionof an organ of a patient. As another example, a recommendation of thespecific action may include a creation of a stoma, as previouslydiscussed by way of example.

Disclosed systems and methods may involve analyzing current and/orhistorical surgical footage to identify features of surgery, patientconditions, and other features to estimate surgical contact force.Exerting too much contact force during a procedure may have adversehealth consequences to a patient. Conversely, insufficient contact forcemay result in suboptimal results for some procedures. Assessing anappropriate level of force to apply in any given surgical situation maybe difficult, resulting in suboptimal outcomes for patients. Therefore,there is a need for unconventional approaches that efficiently,effectively, and in real-time or post-operatively determine surgicalcontact force.

In accordance with the present disclosure, a method for estimatingcontact force on an anatomical structure during a surgical procedure isdisclosed. Contact force may include any force exerted by a surgeon orby a surgical tool on one or more anatomical structures (e.g., a tissue,limb, organ, or other anatomical structure of a patient) during asurgical procedure. The term “contact force” as used herein, refers toany force that may be applied to an anatomical structure, whether thatforce is characterized in a unit of weight (e.g., kilograms or poundsapplied), a unit of force (e.g., Newtons), a pressure applied to an area(e.g., pounds applied per square inch), a tension (e.g., pulling force),or pressure (e.g., pushing force).

Contact force may be applied directly or indirectly in many ways. Forexample, a contact force may be applied through direct contact of asurgeon with an anatomical structure (e.g., applied by a surgeon'shands), or may be applied through a surgical instrument, tool or otherstructure in the surgeon's hands. In cases where the surgeon is asurgical robot, the robot may exert a contact force via a roboticstructure (robotic arm, fingers, graspers) either directly or through atool, instrument or other structure manipulated by the robot.

Contact force may include a normal (i.e., orthogonal) force, a shearforce, and/or a combination of normal and shear forces. More generally,contact force may include any force or pressure applied to any part of apatient's body during a surgery.

Consistent with the present embodiments, estimating contact force mayinclude analyzing images and/or surgical video to generate an estimateof a magnitude of an actual contact force according to a scale. Forceestimation through image analysis may involve an examination of atissue/modality interface to observe an effect on the tissue. Forexample, if the modality is a medical instrument such as forcepspressing against an organ such as a gallbladder, machine visiontechniques applied to the location of force application may revealmovement and/or changes of the organ that is reflective of the forceapplied. Based on historical video footage from prior procedures whereforce application was previously observed, an estimate of the magnitudeof force applied can be made for the current video. The force magnitudeestimate may include a unit of measurement (e.g., pounds, pounds persquare inch, Newtons, kilograms, or other physical units) or may bebased on a relative scale. A relative scale may include a categoricalscale, a numeric scale, and/or any other measure. A categorical scalemay reflect a level of force (e.g., a scale including multiple levelssuch as a high force, a medium force, a low force, or any other numberof levels). A contact force may be estimated according to a numericalscale such as a scale of 1-10. Moreover, the force may be estimated atdiscrete points in time or may be estimated continuously. In someembodiments, an estimate of a contact force may include an estimate of acontact location, a contact angle, and/or an estimate of any otherfeature of contact force.

In some embodiments, a method for estimating contact force on ananatomical structure may include receiving, from at least one imagesensor in an operating room, image data of a surgical procedure. Animage sensor may include a camera and/or any other image capture device.An image sensor may be configured to collect image data and/or videodata and may be positioned anywhere in any operating room, such as, forexample, above a patient or within a patient (e.g., in an intracorporealcavity). Image data may include surgical video, video clips, videofootage, image frames, continuous video and/or any other informationderived from video. For example, image data may include pixel data,color data, saturation data, and/or any other data representing animage, regardless of storage format. Image data may include time data(e.g., a time an image was captured by a sensor), location data,information relating to a surgical procedure (e.g., a patientidentifier, a name of a surgical procedure) and/or any other metadata.In some embodiments, image data of a surgical procedure may be collectedby an image sensor in an operating room and stored in a data structure(e.g., a data structure of FIG. 17A) in, near, or even remote from theoperating room. While the force estimation may occur in real time, itmay also be estimated in non-real time, such as when the data isretrieved from a data structure.

In some embodiments, a method for estimating contact force on ananatomical structure may include analyzing received image data todetermine an identity of an anatomical structure reflected in imagedata. Analyzing received image data may include any method of imageanalysis, consistent with the present embodiments. Some non-limitingexamples of algorithms for identifying anatomical structures in imagesand/or videos are described above. Analyzing received image data mayinclude, for example, methods of object recognition, imageclassification, homography, pose estimation, motion detection, and/orother image analysis methods. Analyzing received image data may includeartificial intelligence methods including implementing a machinelearning model trained using training examples, consistent withdisclosed embodiments. For example, received image data may be analyzedusing a machine learning model trained using training examples to detectand/or identify an anatomical structure, for example as described above.For example, received image data may be analyzed using an artificialneural network configured to detect and/or identify an anatomicalstructure from images and/or videos. Training examples may include imagedata labeled or otherwise classified as depicting an anatomicalstructure (e.g., images classified as depicting a pancreas).

In some embodiments, a method for estimating contact force on ananatomical structure may include analyzing received image data todetermine a condition of anatomical structure. Generally, a condition ofan anatomical structure may refer to any information that indicates astate or characteristic of an anatomical structure. For example, acondition may reflect whether an anatomical structure is normal,abnormal, damaged, leaking, hydrated, dehydrated, oxygenated, retracted,enlarged, shrunken, present, absent, and/or any other assessment. Acondition may include a measure of a vitality of an anatomicalstructure, a level of oxygenation, a level of hydration, a level ofdistress, and/or a measure of any other state of an anatomicalstructure. In one example, a condition of an anatomical structure may berepresented as a vector of numerical values corresponding to a point ina mathematical space. In some examples, a machine learning model may betrained using training examples to identify conditions of anatomicalstructures from images and/or videos, and the trained machine learningmodel may be used to analyze the received image data and determine thecondition of the anatomical structure. An example of such trainingexample may include an image and/or a video of an anatomical structure,together with a label indicating the condition of the anatomicalstructure.

In some embodiments, an analysis may determine a condition based on acharacteristic of an anatomical structure that indicates a condition. Asa non-limiting example, an analysis may determine a color of a tissue, atexture of an anatomical structure, a heart rate, a lung capacity,and/or any other characteristic of an anatomical structure. In someembodiments, a recommendation may be based on a characteristic reflectedin sensor data such as heart rate monitor data, brain activity data,temperature data, blood pressure data, blood flow data, leakage data,and/or any other health data. Such characteristics of an anatomicalstructure may indicate a condition of the anatomical structure and maybe correlated with known conditions. For example, reduced brain activitymight be indicative of a vessel blockage or increased cranial pressuremight be indicative of a brain hemorrhage. Such correlations may bestored a data structure such as a data structure of FIG. 17A).

In some embodiments, a method for estimating contact force on ananatomical structure may include selecting a contact force thresholdassociated with an anatomical structure. A contact force threshold mayinclude a minimum or maximum contact force. In some embodiments,selecting a contact force threshold may be based on informationindicating a likely outcome associated with applying forces above orbelow a threshold. Selecting a contact force threshold may be based ondata indicating a recommended contact force (e.g., a maximum safe forceor a minimum effective force). For example, selecting a contact forcethreshold may be based on a table of anatomical structures includingcorresponding contact force thresholds. A table may include indicationsof conditions of anatomical structures. In some embodiments, a selectedcontact force threshold may be based on a determined condition of ananatomical structure. For example, a selected contact force thresholdmay increase or decrease based on information indicating an anatomicalstructure is leaking, has a particular color, has a particular level ofretraction, and/or any other condition. In another example, in responseto a first determined condition of the anatomical structure, a firstcontact force threshold may be selected, and in response to a seconddetermined condition of the anatomical structure, a second contact forcethreshold may be selected, the second contact force threshold may differfrom the first contact force threshold. In yet another example, thedetermined condition of the anatomical structure may be represented as avector (as described above), and the contact force threshold may becalculated using a function of the vector representation of thedetermined condition. In some examples, a selected contact forcethreshold may be a function of a type of the contact force (such astension, compression, and so forth). For example, in response to a firsttype of contact force, the selected contact force threshold may have afirst value, and in response to a second type of contact force, theselected contact force threshold may have a second value, the secondvalue may differ from the first value.

In some embodiments, a contact force threshold may be associated with atension level (i.e., a level of force that pulls on an anatomicalstructure) or a level of retraction. Retraction may involve movement,traction, and/or counter-traction of tissues to expose tissue, organ,and/or other anatomical structure for viewing by a surgeon. In someembodiments, a contact force threshold may be associated with a pressurelevel (e.g., an amount of contact force that pushes on an anatomicalstructure) and/or a compression level. A compression level may include adegree or amount of compression of an anatomical structure (e.g., areduction in size of an anatomical structure due to contact force).

Consistent with the present embodiments, selecting a contact force maybe based on data relating to a manner of contact between an anatomicalstructure and a medical instrument. For example, in some embodiments,selecting a contact force threshold may be based on a location ofcontact between an anatomical structure and a medical instrument, assome regions of anatomical structures may have greater force sensitivitythan others. A location may be determined by analyzing received imagedata, consistent with disclosed embodiments. Thus, a selected contactforce threshold may be higher at one location of contact between ananatomical structure and a medical instrument than at another. Selectinga contact force threshold may also be based on an angle of contactbetween an anatomical structure and a medical instrument. An angle ofcontact may be determined by analyzing image data to identify theincidence angle between an anatomical structure and a medicalinstrument. For example, pose estimation algorithms may be used toanalyze the image data and determining a pose of the anatomicalstructure and/or a pose of the medical instrument, and an angle betweenthe anatomical structure and the medical instrument may be determinedbased on the determined poses. In another example, a machine learningalgorithm may be trained using training examples to determine anglesbetween anatomical structures and medical instruments, and the trainedmachine learning model may be used to analyze the image data anddetermine the angle between the anatomical structure and the medicalinstrument. An example of such training example may include an imagedepicting an anatomical structure and a medical instrument, togetherwith a label indicating the angle between the anatomical structure andthe medical instrument. In some examples, a selected contact forcethreshold may be a function of a contact angle related to the contactforce. For example, in response to a first contact angle, the selectedcontact force threshold may have a first value, and in response to asecond contact angle, the selected contact force threshold may have asecond value, the second value may differ from the first value.

In some embodiments, selecting a contact force threshold may includeimplementing and/or using a model (e.g., a statistical model and/or amachine learning model). For example, selecting a contact forcethreshold may include providing a condition of an anatomical structureto a regression model as an input and selecting a contact forcethreshold based on an output of a regression model. In some embodiments,a regression model may be fit to historical data comprising contactforces applied to anatomical structures with corresponding conditionsand surgical outcomes.

In some embodiments, selecting a contact force threshold may includeusing a machine learning model trained using training examples to selecta contact force threshold. For example, a machine learning model may betrained using training examples to select contact force thresholds basedon input data. Such input data may include image data of a surgicalprocedure, image data depicting an anatomical structure, a type of asurgical procedure, a phase of a surgical procedure, a type of action, atype of an anatomical structure, a condition of an anatomical structure,a skill level of a surgeon, a condition of a patient, and so forth. Anexample of such training example may include a sample input datatogether with a label indicating the desired contact force threshold. Inone example, the desired contact force threshold may be selected basedon known medical guidelines. In another example, the desired contactforce threshold may be selected manually. In yet another example, thedesired contact force threshold may be selected based on an analysis ofcorrelations of applied contact force and outcome in historical cases orin a define subset of a group of historical cases, for example to selecta contact force threshold that is highly correlated with positiveoutcome (for example, ensure positive outcome according to historicaldata, ensure positive outcome in a selected ratio of cases according tohistorical data, and so forth). Further, in some examples, the trainedmachine learning model may be used to analyze such input datacorresponding to a particular case (such as a particular surgicalprocedure, a particular phase of a surgical procedure, a particularaction in a surgical procedure, a particular surgeon, a particularpatient, a particular anatomical structure, etc.) and select the contactforce threshold. For example, the trained machine learning model may beused to analyze the image data of the surgical procedure and/or thedetermined identity of the anatomical structure and/or the determinedcondition of the anatomical structure and/or characteristics of acurrent state of the surgical procedure to select the contact forcethreshold.

In some embodiments, a machine learning model may be trained usingtraining examples to determine contact properties (such as contactlocation, a contact angle, a contact force) from images and/or videos,and the trained machine learning model may be used to analyze the videofootage and determine the properties of an actual contact occurring inthe surgical procedure, such as the actual contact location, the actualcontact angle, the actual contact force, and so forth. An example of atraining example may include image data depicting a particular contacttogether with a label indicating properties of the particular contact,such as a contact location, a contact angle, a contact force, and soforth. For example, a training example may include measurements ofcontact force collected using a sensor (e.g., a sensor embedded in amedical instrument). In another example, a training example may includeestimates of contact force included in a medical record (e.g., anestimate of contact force stored in a record, an estimate based onsensor data or a surgeon's opinion).

In some embodiments, selecting a contact force threshold may be based onone or more actions performed by a surgeon. For example, a method mayinclude analyzing image data to identify actions performed by a surgeon(e.g., a human or a surgical robot), for example using actionrecognition algorithms. In one example, the selected contact forcethreshold may be based on historical data correlating one or moreactions performed by a surgeon, contact forces, and outcomes. Forexample, a contact force threshold that is highly correlated withpositive outcome may be selected (for example, that ensures positiveoutcome according to historical data, that ensures positive outcome in aselected ratio of cases according to historical data, and so forth). Inone example, a data structure may specify the contact force thresholdsfor different actions. In one example, the contact force threshold maybe based on a level of skill of a surgeon, consistent with disclosedembodiments.

In some embodiments, a method for estimating contact force on ananatomical structure may include receiving an indication of actualcontact force on an anatomical structure. An indication of an actualcontact force may be associated with a contact between a surgeon (e.g.,a human or robotic surgeon) and an anatomical structure, directly orindirectly. For example, an actual contact force may be associated witha contact between a medical instrument and an anatomical structure(e.g., between an anatomical structure and a reactor, a scalpel, asurgical clamp, a drill, a bone cutter, a saw, scissors, forceps, and/orany other medical instrument). In some embodiments, an actual force maybe associated with a tension level, a level of retraction, a pressurelevel, and/or a compression level. An indication may include an estimateof contact force, including a level of contact, consistent withdisclosed embodiments. More generally, an indication of an actual forcemay include any indication of any contact force, as described herein,that is applied during a surgical event. In one example, the indicationof the actual contact force may include at least one of an indication ofa contact angle, an indication of a magnitude or level of the contactforce, and indication of a type of the contact force, and so forth.

In some embodiments, an indication of actual contact force may beestimated based on an image analysis of image data. An image analysis ofimage data to estimate an indication of contact force may include anymethod of image analysis as disclosed herein. In some embodiments, anindication of contact force may be based on image analysis methods thatassociate a contact force with a change in an anatomical structure(e.g., a deformation of an anatomical structure), a position of asurgeon or surgical instrument, a motion of a surgeon and/or a surgicalinstrument, and/or any other feature of a surgical event. In someembodiments, an indication of actual contact force may be estimatedusing a regression model fit to historical data associating a contactforce with a feature of surgical event. Also, an indication of actualcontact force may be estimated using a machine learning model, forexample as described above.

In some embodiments, an indication of actual contact force may be basedon sensor data that directly or indirectly measures force. For example,an actual force may be based on a force sensor that measures force at alocation of contact between a medical instrument or surgical robot andan anatomical structure (e.g., a force sensor embedded in a medicalinstrument or robot). In an exemplary embodiment, an indication ofactual contact force may be received from a surgical tool or othermedical instrument. Similarly, an indication of actual contact force maybe received from a surgical robot.

In some embodiments, a method for estimating contact force on ananatomical structure may include comparing an indication of actualcontact force with a selected contact force threshold, which may includedetermining whether an actual contact force exceeds or fails to exceed aselected contact force threshold. Comparing an indication of actualcontact force with a selected contact force threshold may includecalculating a difference, a ratio, a logarithm, and/or any otherfunction of an actual contact force and a selected contact forcethreshold.

In some embodiments, a method for estimating contact force on ananatomical structure may include outputting a notification based on adetermination that an indication of actual contact force exceeds aselected contact force threshold. Outputting a notification may includetransmitting a recommendation to a device, displaying a notification atan interface, playing a sound, providing haptic feedback, and/or anyother method of notifying an individual of excessive force applied. Anotification may be output to a device in an operating room, to a deviceassociated with a surgeon (e.g., a human surgeon and/or a surgicalrobot), and/or to any other system. For example, outputting anotification may include transmitting a notification to a computer, amobile device, an external device, a surgical robot, and/or any othercomputing device. In another example, outputting a notification mayinclude logging the notification in a file.

In some embodiments, a notification may include information specifyingthat a contact force has exceeded or failed to exceed a selected contactforce threshold. In some embodiments, a notification may includeinformation relating to a selected contact force and/or an estimate ofan actual contact force, including an indication of a contact angle, amagnitude of a contact force, a contact location, and/or otherinformation relating to a contact force.

In some examples, notifications of different intensity (i.e., severityor magnitude) may be provided according to an indication of actualforce. For example, outputting a notification may be based on adifference between an indication of actual force and a selected forcethreshold or a comparison of an indication of actual force with aplurality of thresholds. A notification may be based on a level ofintensity of an actual force or an intensity of a difference between anactual force and a selected force threshold. In some embodiments, anotification may include information specifying a level of intensity.

Consistent with the present embodiments, a notification may be output inreal time during a surgical procedure, such as to provide warming to asurgeon conducting a surgical procedure. In some embodiments, anotification may include an instruction to a surgical robot to vary aforce application. As an illustrative example, a notification mayinclude an instruction to after a magnitude, angle, and/or location of acontact force.

In some embodiments, a method for estimating contact force on ananatomical structure may include determining from received image datathat a surgical procedure is in a fight mode, where extraordinarymeasures may be required. In such circumstances, typical contact forcethresholds may be suspended. Determining from received image data that asurgical procedure may be in a fight mode may include using a method ofimage analysis, as disclosed herein. For example, certain physiologicalresponses and/or surgical activities depicted in the video may indicatethat the surgical procedure is in fight mode. A fight mode determinationmay include using a statistical model (e.g., a regression model) and/ora machine learning model, such as a model trained to recognize fightmode using historical examples of surgical video classified as depictingportions of surgeries that are and are not in a fight mode. In someembodiments, a notification may be suspended during a fight mode. Forexample, outputting a notification may be delayed indefinitely or atleast until a determination is made that a surgical procedure may be notin a fight mode. In some embodiments, outputting a notification may bedelayed for a predetermined time period (e.g., a number of minutes orany other time period). In other examples, the type of the outputtednotifications may be determined based on whether the patient undergoingthe surgical procedure is in a fight mode. In some examples, the contactforce thresholds may be selected based on whether the patient undergoingthe surgical procedure is in a fight mode.

In some embodiments, a method for estimating contact force on ananatomical structure may include determining from received image datathat a surgeon may be operating in a mode ignoring contact forcenotifications. A contact force notification may include a notificationincluding information relating to a contact force (e.g., an actualcontact force and/or a selected contact force threshold). In someembodiments, a determination that a surgeon may be operating in a modeignoring contact force notifications may include analyzing one or moreindications of actual contact force following one or more contact forcenotifications. For example, embodiments may include determining whetherone more actual contact force indications exceed or fails to exceed aselected contact force threshold following output of one or more contactforce notifications. Determining from received image data that a surgeonmay be operating in a mode ignoring contact force notifications mayinclude using a method of image analysis, and may include using astatistical model (e.g., a regression model) and/or a machine learningmodel. Such machine learning models may be trained to determine that asurgeon may be operating in a mode ignoring contact force notificationsusing historical examples of surgical video classified as surgeons thatare and are not ignoring contact force notifications.

Embodiments may include suspending (delaying) at least temporarily,further contact force notifications based on a determination that asurgeon may be operating in a mode ignoring contact force notifications.In some embodiments, contact force notifications may resume following apredetermined time period (e.g., a number of minutes or any other timeperiod).

FIG. 30 is a flowchart illustrating an exemplary process 3000 forestimating contact force on an anatomical structure, consistent with thedisclosed embodiments. Process 3000 may be performed by at least oneprocessor, such as one or more microprocessors. In some embodiments,process 3000 is not necessarily limited to the steps illustrated, andany of the various embodiments described herein may also be included inprocess 3000. As one of skill in the art will appreciate, steps ofprocess 3000 may be performed by a system including, for example,components of system 1401. In some embodiments, a non-transitorycomputer readable medium including instructions that, when executed byat least one processor, cause the at least one processor to executeoperations for estimating contact force on an anatomical structureaccording to process 3000. In some embodiments, process 3000 may beperformed in real time during a surgical procedure.

At step 3002, the process may include receiving, from at least one imagesensor in an operating room, image data of a surgical procedure, aspreviously described through various examples. An image sensor may beplaced anywhere in any operating room, and image data may include anyvideo data, data representing an image, and/or metadata.

At step 3004, the process may include analyzing the received image datato determine an identity of an anatomical structure and to determine acondition of the anatomical structure as reflected in the image data,consistent with disclosed embodiments, as describe previously throughexamples. Analyzing received image data may include any method of imageanalysis, as previously described, and a condition of an anatomicalstructure may refer to any information that indicates a state orcharacteristic of an anatomical structure. As discussed previously,analyzing the received image data may include using a machine learningmodel trained using training examples to determine a condition of ananatomical structure in image data.

At step 3006, the process may include selecting a contact forcethreshold associated with the anatomical structure, the selected contactforce threshold being based on the determined condition of theanatomical structure. As previously discussed in greater detail,selecting a contact force threshold may be based on data indicating arecommended contact force (e.g., a maximum safe force or a minimumeffective force). Selecting a contact force threshold may be based on alocation and/or angle of contact force and may include implementing amodel (e.g., a statistical model such as a regression model and/or amachine learning model). Further, a table of anatomical structuresincluding corresponding contact force thresholds may be used as part ofselecting a contact force threshold. A contact force threshold may beassociated with a tension level or a compression level. In someexamples, selecting a contact force threshold may include using amachine learning model trained using training examples to select acontact force threshold. Further, selecting a contact force thresholdmay be based on one or more actions performed by a surgeon. Othernon-limiting examples of the selection of a contact force threshold aredescribed above.

At step 3008, the process may include receiving an indication of actualcontact force on the anatomical structure (for example, as discussedpreviously), such as with a force associated with a contact between amedical instrument and an anatomical structure. An actual force may beassociated with a tension level, a level of retraction, a pressurelevel, and/or a compression level. An indication of actual contact forcemay be estimated based on an image analysis of image data. An indicationof actual contact force may be based on sensor data that directly orindirectly measures force. In some embodiments, an indication of actualcontact force may be estimated based on an image analysis of image dataand/or may be an indication of an actual contact force received from asurgical tool, surgical robot, or other medical instrument.

At step 3010, the process may include comparing the indication of actualcontact force with the selected contact force threshold, as discussedpreviously. Comparing an indication of actual contact force with aselected contact force threshold may include calculating a difference, aratio, a logarithm, and/or any other function of an actual contact forceand a selected contact force threshold.

At step 3012, the process may include outputting a notification based ona determination that the indication of actual contact force exceeds theselected contact force threshold, as previously described. Outputting anotification may be performed in real time during an ongoing surgicalprocedure. For example, outputting a notification may include providinga real time warming to a surgeon conducting a surgical procedure or aninstruction to a surgical robot.

Disclosed systems and methods may involve analyzing current and/orhistorical surgical footage to identify features of surgery, patientconditions, and other features to update a predicted surgical outcome.Over the course of a surgical procedure, conditions may change, orevents may transpire that change a predicted outcome of the surgicalprocedure. Conventional approaches to performing surgery may lackdecision support systems to updated predicted outcomes during real timebased on surgical events as they occur. As a result, surgeons may beunaware of likely surgical outcomes and thereby may be unable to performactions that may improve outcomes or that may avoid worsening outcomes.Therefore, aspects of the current disclosure relate to unconventionalapproaches that efficiently, effectively, and in real time updatepredicted surgical outcomes.

In accordance with the present disclosure, a systems, methods andcomputer readable media may be provided for updating a predicted outcomeduring a surgical procedure is disclosed. For example, image data may beanalyzed to detect changes in a predicted outcome, and a remedial actionmay be communicated to a surgeon. A predicted outcome may include anoutcome that may occur with an associated confidence or probability(e.g., a likelihood). For example, a predicted outcome may include acomplication, a health status, a recovery period, death, disability,internal bleeding, hospital readmission after the surgery, and/or anyother surgical eventuality. In some embodiments, a predicted outcomeincludes a score, such as a lower urinary tract symptom (LUTS) outcomescore. More generally, a predicted outcome may include any healthindicator associated with a surgical procedure.

In some embodiments, a predicted outcome may include a likelihood ofhospital readmission, such as a likelihood of a hospital readmission ofthe patient undergoing the surgical procedure within a specified timeinterval after the patient is been discharged from the hospitalfollowing the surgical procedure. Hospital readmission may be based on ahealth condition related to a surgical procedure, or may be based onother factors. For example, a hospital readmission may arise due to apost-operative complication (e.g., swelling, bleeding, an allergicreaction, a ruptured suture, and/or any other complication). In someembodiments, a likelihood of hospital readmission may be determinedbased on an analysis of image data (e.g., using image analysis methodsas described herein). Further, in some embodiments, a likelihood ofhospital readmission may be determined based on information of a patientundergoing a surgical procedure. For example, a likelihood of hospitalreadmission may be based on a patient characteristic (e.g., an age, aprior health status, a family history, a vital sign, and/or otherpatient-related data). Hospital readmission may be defined for differenttime intervals (e.g., readmission within 24 hours, within a week, withina month, or within another time period).

In some embodiments, a predicted outcome may be based on at least onemodel, such as statistical model and/or a machine learning model. Forexample, a predicted outcome may be based on statistical correlationsbetween information associated with a surgical procedure (e.g., patientcharacteristic and/or a surgical event) and historical outcomes. Apredicted outcome may be generated by a machine learning model trainedto associate outcomes with information associated with a surgicalprocedure (e.g., patient characteristic and/or a surgical event) usingtraining examples (for example, using training examples based onhistorical data).

Disclosed embodiments may include receiving, from at least one imagesensor arranged to capture images of a surgical procedure, image dataassociated with a first event during a surgical procedure, consistentwith disclosed embodiments. Image data associated with a first event mayinclude still images, image frames, clips and/or video-related dataassociated with a surgical procedure. A first event may include anysurgical event, consistent with disclosed embodiments. In anillustrative embodiment, a first event may include an action performedby a surgeon (e.g., a human or robotic surgeon). In another example, afirst event may include a physiological response to an action. In yetanother example, a first event may include a change in a condition of ananatomical structure. Some other non-limiting examples of such surgicalevents are described above. Image data associated with a first event maybe received in memory and/or a data structure, as described by way ofexample herein.

An image sensor may include any image sensor as also described herein(e.g., a camera or other detector). In some embodiments, an image sensormay be positioned in an operating room. For example, an image sensor maybe positioned above a patient undergoing a surgical procedure or withina patient undergoing a surgical procedure (e.g., an intracavitarycamera).

Disclosed embodiments may include determining, based on received imagedata associated with a first event, a predicted outcome associated witha surgical procedure, consistent with disclosed embodiments. A predictedoutcome may include any health outcome associated with a surgicalprocedure, as described above. For example it may include an eventualitythat is correlated in some way to the first event. The prediction may bebinary (e.g., likely to result in a rupture vs. not likely to result ina rupture), or it may provide a relative confidence or probability(e.g., percent chance of rupture; chance of rupture on a scale of 1-5;and so forth). A determined predicted outcome may include a scorereflecting a property of an outcome such as a post-operative healthstatus (e.g., a LUTS outcome score). A predicted outcome may beassociated with a confidence or probability.

A first event, as mentioned in the preceding paragraph, may include anyintraoperative occurrence. For example, a first event may include anaction performed by a surgeon, a change in a patient characteristic, achange in a condition of an anatomical structure, and/or any othercircumstance. In some embodiments, at least one time point associatedwith a first event may be received, such that in addition to anindicator of the event itself, an indicator of the time the eventoccurred is also received. The time point may coincide with a counter ona video timeline, or might include any other marker or indicator ofreflecting an absolute or relative time when an event occurred.

Some embodiments may involve identifying an event, such as a firstevent. Such identification may be based, for example, on detection of amedical instrument, an anatomical structure, and/or an interactionbetween a medical instrument and an anatomical structure. The detectioncan occur using video analysis techniques described throughout thisdisclosure. For example, the event may be identified by analyzing theimage data using a machine learning model as described above.

In some embodiments, determining a predicted outcome may includeidentifying an interaction between a surgical tool and an anatomicalstructure and determining a predicted outcome based on the identifiedinteraction. For example, the interaction between the surgical tool andthe anatomical structure may be identified by analyzing the image data,for example as described above. Further, in one example, in response toa first identified interaction, a first outcome may be predicted, and inresponse to a second identified interaction, a second outcome may bepredicted, the second outcome may differ from the first outcome. Inanother example, a machine learning model may be trained using trainingexamples to predict outcome of surgical procedures based on interactionsbetween surgical tools and anatomical structures, and the trainedmachine learning model may be used to predict the outcome based on theidentified interaction. An example of such training example may includean indication of an interaction between a surgical tool and ananatomical structured, together with a label indicating the desiredpredicted outcome. The desired predicted outcome may be based on ananalysis of historical data, based on user input (such as expertopinion), and so forth.

In some embodiments, determining a predicted outcome may be based on askill level of a surgeon depicted in image data, such as data previouslystored in a data structure. The surgeon's level of skill may bedetermined based on an analysis of image data, for example as describedabove. For example, a face recognition algorithm may be applied to imagedata to identify a known surgeon, and a corresponding level of skill maybe retrieved from a data structure, such as a database. In someembodiments, a level of skill of a surgeon may be determined based on asequence of events identified in image data (e.g., based on a length oftime to perform one or more actions, based on a patient responsedetected in image data during surgery, and/or based on other informationindicating a level of skill of a surgeon). In one example, in responseto a first determined skill level, a first outcome may be predicted, andin response to a second determined skill level, a second outcome may bepredicted, the second outcome may differ from the first outcome. Inanother example, a machine learning model may be trained using trainingexamples to predict outcome of surgical procedures based on skill levelsof surgeons, and the trained machine learning model may be used topredict the outcome based on the determined skill level. An example ofsuch training example may include an indication of a skill level of asurgeon, together with a label indicating the desired predicted outcome.The desired predicted outcome may be based on an analysis of historicaldata, based on user input (such as expert opinion), and so forth.

Determining a predicted outcome may also be based, in some instances, ona condition of an anatomical structure depicted in image data. Forexample, a predicted outcome may be determined based on historicaloutcomes correlated with organ condition. Complications with organs inpoor condition might, for example, be greater than with organs in goodcondition. A condition of an anatomical structure may be determined, insome embodiments, based on an analysis of image data as describedthroughout this disclosure. The anatomical structure's condition may betransient or chronic and/or include a medical condition, such as acondition being treated by a surgical procedure or a separate medicalcondition. A condition of an anatomical structure may be indicated bycolor, texture, size, level of hydration, and/or any other observablecharacteristic. In one example, in response to a first determinedcondition of the anatomical structure, a first outcome may be predicted,and in response to a second determined condition of the anatomicalstructure, a second outcome may be predicted, the second outcome maydiffer from the first outcome. In another example, a machine learningmodel may be trained using training examples to predict outcome ofsurgical procedures based on conditions of anatomical structures, andthe trained machine learning model may be used to predict the outcomebased on the determined condition of the anatomical structure. Anexample of such training example may include an indication of acondition of an anatomical structure, together with a label indicatingthe desired predicted outcome. The desired predicted outcome may bebased on an analysis of historical data, based on user input (such asexpert opinion), and so forth.

Additionally or alternatively, a predicted outcome may be determinedbased on an estimated contact force on an anatomical structure. Forexample, an excessive force applied to the anatomical structure mayrender a favorable outcome less likely. For example, the contact forcemay be estimated by analyzing the image data, for example as describedabove. In another example, the contact force may be received from asensor, for example as described above. In one example, in response to afirst estimated contact force, a first outcome may be predicted, and inresponse to a second estimated contact force, a second outcome may bepredicted, the second outcome may differ from the first outcome. Inanother example, a machine learning model may be trained using trainingexamples to predict outcome of surgical procedures based on contactforces on anatomical structures, and the trained machine learning modelmay be used to predict the outcome based on the estimated contact force.An example of such training example may include an indication of acontact force, together with a label indicating the desired predictedoutcome. The desired predicted outcome may be based on an analysis ofhistorical data, based on user input (such as expert opinion), and soforth.

Determining a predicted outcome may be performed in various ways. It mayinclude using a machine learning model trained to determine predictedoutcomes based on historical surgical videos and information indicatingsurgical outcome corresponding to historical surgical videos. Forexample, received image data of a first event may be analyzed using anartificial neural network configured to predict outcome of surgicalprocedures from images and/or videos. As another example, determining apredicted outcome may include identifying a first event based onreceived image data and applying a model (e.g., a statistical model or amachine learning model) to information relating to a first event topredict an outcome. Such a model may receive inputs, includinginformation relating to a first event (e.g., an identifier of a firstevent, a duration of a first event, and/or other property of a firstevent such as a surgical contact force) and/or information relating to asurgical procedure (e.g., a patient characteristic, a level of skill ofa surgeon, or other information). Based on inputs such as the examplesprovide above, the system may return a predicted outcome as an output.

Disclosed embodiments may include receiving, from at least one imagesensor arranged to capture images of a surgical procedure, image dataassociated with a second event during a surgical procedure, consistentwith disclosed embodiments. A second event may occur after the firstevent and may be different from the first event. At least one time pointassociated with a second event may be received. The image sensor forcapturing data associated with the second event may be the same as ormay be different from the image sensor used to capture data associatedwith the first event.

Disclosed embodiments may include determining, based on received imagedata associated with a second event, a change in a predicted outcome,causing a predicted outcome to drop below a threshold. For example,using any of the methods described above to determine a predictedoutcome, a new predicted outcome may be determined and compared to apreviously determined predicted outcome (such as the predicted outcomedetermined based on the received image data associated with the firstevent) to thereby determine a change in a predicted outcome. In anotherexample, the new predicted outcome may be determined based on apreviously determined predicted outcome (such as the predicted outcomedetermined based on the received image data associated with the firstevent) and an analysis of the received image data associated with thesecond event. For example, a machine learning model may be trained usingtraining examples to determine new predicted outcomes based on previouspredicted outcomes and images and/or videos, and the trained machinelearning model may be used to analyze the previously determinedpredicted outcome and the received image data associated with the secondevent to determine the new predicted outcome. An example of suchtraining example may include a previously determined predicted outcomeand an image data depicting an event, together with a label indicatingthe new predicted outcome. In another example, a Markov model may beused to update the previously determined predicted outcome and obtainthe new predicted outcome, where the transitions in the Markov model maybe based on values determined by analyzing the received image dataassociated with the second event. As discussed, a predicted outcome mayinclude a probability, confidence, and/or score reflecting a property ofan outcome such as a post-operative health status (e.g., a LUTS outcomescore). Determining a change in a predicted outcome may involve a changein such a confidence, probability or score. In some examples, a changein a predicted outcome may be determined without calculating a newpredicted outcome. For example, a machine learning model may be trainedusing training examples to determine a change in predicted outcomesbased on previous predicted outcomes and images and/or videos, and thetrained machine learning model may be used to analyze the previouslydetermined predicted outcome and the received image data associated withthe second event to determine an occurrence of a change in a predictedoutcome. An example of such training example may include a previouslydetermined predicted outcome and an image data depicting an event,together with a label indicating whether the predicted outcome havechanged in response to the second event.

In some embodiments, a change in a confidence, probability, and/or scoremay cause a predicted outcome to drop below a threshold (e.g., athreshold confidence, a threshold probability, a threshold score). Suchthreshold may be automatically generated using artificial intelligencemethods, may be determined based on user input, and so forth. Athreshold may correspond to a negative outcome (such as a hospitalreadmission, a complication, death, or any undesirable eventuality), orto a positive outcome.

In some illustrative embodiments, determining a change in a predictedoutcome may be based on elapsed time between two markers. For example, aduration between an incision and suturing that exceeds a threshold mayserve as an indicator of an increased likelihood of infection. Forexample, in response to a first elapsed time, a change in the predictedoutcome may be determined, and in response to a second elapsed time, nochange in the predicted outcome may be determined.

In some examples, two or more variables may be correlated to eitherpositive or negative outcomes, for example using statistical methods,using machine learning methods, and so forth. The variables may beendless. Such variables may relate to the condition of the patient, thesurgeon, the complexity of the procedure, complications, the tools used,the time elapsed between two or more events, or any other variables orcombination of variables that may have some direct or indirect impact onpredicted outcome. One such variable may be fluid leakage (e.g., amagnitude, duration, or determined source). For example, determining achange in a predicted outcome may be based on a magnitude of bleeding. Afeature of a fluid leakage event (e.g., a magnitude of bleeding, asource of bleeding) may be determined based on an analysis of imagedata.

Disclosed embodiments may include determining a skill level of a surgeondepicted in image data, and determining a change in a predicted outcomemay be based on the skill level. For example, a determining a change ina predicted outcome may be based on an updated estimate of a level ofskill of a surgeon (e.g., an image analysis may determine that a surgeonhas made one or more mistakes, causing an estimate of level of skill todecrease). As another example, a previously determined predicted outcomemay be based on a level of skill of a first surgeon and a change in apredicted outcome may be based on a level of skill of a second surgeonwho steps in to assist. A level of skill may be determined in variousways, as described herein (e.g., via an image analysis as describedabove and/or by retrieving a level of skill from a data structure).

By way of additional examples, determining a change in a predictedoutcome may be based on one or more changes in color, texture, size,condition, or other appearance or characteristic of at least part of ananatomical structure. Examples of conditions of anatomical structuresthat may be used for outcome prediction may vitality, a level ofoxygenation, a level of hydration, a level of distress, and/or any otherindicator of the state of the anatomical structure.

A condition of an anatomical structure may be determined in a variety ofways, such as through a machine learning model trained with examples ofknown conditions. In some embodiments, an object recognition modeland/or an image classification model may be trained using historicalexamples and implemented to determine a condition of an anatomicalstructure. Training may be supervised and/or unsupervised. Some othernon-limiting examples of methods for determining conditions ofanatomical structures are described above.

Embodiments may include a variety of ways of determining a predictedoutcome based on a condition of an anatomical structure and/or any otherinput data. For example, a regression model may be fit to historicaldata that include conditions of anatomical structures and outcomes. Moregenerally, using historical data, a regression model may be fit topredict an outcome based on one or more of a variety of input data,including a condition of an anatomical structure, a patientcharacteristic, a skill level of a surgeon, an estimated contact force,a source of fluid leakage, an extent of fluid leakage characteristic,and/or any other input data relating to a surgical procedure. An outcomemay be predicted based on other known statistical analysis including,for example, based on correlations between input data relating to asurgical procedure and outcome data.

Disclosed embodiments may include accessing a data structure ofimage-related data based on prior surgical procedures, consistent withdisclosed embodiments. Accessing may include reading and/or writing datafrom a data structure. In some embodiments, this may be accomplishedusing a data structure such as is presented in FIG. 17 or a datastructure such as is presented in FIG. 6 . Image related data mayinclude any data derived directly or indirectly from images. This datamay include, for example, patient characteristics, surgeoncharacteristics (e.g., a skill level), and/or surgical procedurecharacteristics (e.g., an identifier of a surgical procedure, anexpected duration of a surgical procedure). Image related data mayinclude correlations or other data describing statistical relationshipsbetween historical intraoperative surgical events and historicaloutcomes. In some embodiments, a data structure may include datarelating to recommended actions, alternative courses of action, and/orother actions that may change a probability, likelihood, or confidenceof a surgical outcome. For example, a data structure may includeinformation correlating a break from a surgical procedure with animproved outcome. Depending on implementation, a data structure mayinclude information correlating a skill level of a surgeon, a requestfor assistance from another surgeon, and outcomes. Similarly, a datastructure may store relationships between surgical events, actions(e.g., remedial actions), and outcomes. While a host of correlationmodels may be used for prediction as discussed throughout thisdisclosure, exemplary predictive models may include a statistical modelfit to historical image-related data (e.g., information relating toremedial actions) and outcomes; and a machine learning models trained topredict outcomes based on image-related data using training data basedon historical examples.

Disclosed embodiments may include identifying, based on accessedimage-related data, a recommended remedial action. For example, arecommended remedial action may include a recommendation for a surgeonto use a different tool or procedure; administer a drug, requestassistance from another surgeon, make a revision to a surgicalprocedure, take a break from a surgical procedure (for example, toincrease alertness), and/or to undertake any other action that mightimpact outcome. When a recommended remedial action includes a suggestionto request assistance, the suggestion may recommend that a surgeon besummoned with a higher or different level of experience than theoperating surgeon. A remedial action that suggests a revision to asurgical procedure may include a suggestion to perform additionalactions not previously part of a surgical procedure, or to avoid certainexpected actions.

Identifying a remedial action may be based on an indication, derived atleast in part from image-related data, that a remedial action may belikely to raise a predicted outcome above a threshold. For example, adata structure may contain correlations between historical remedialactions and predicted outcomes, and a remedial action may be identifiedbased on the correlations. In some embodiments, identifying a remedialaction may include using a machine learning model trained to identifyremedial actions using historical examples of remedial actions andsurgical outcomes. Training may be supervised or unsupervised. Forexample, the machine learning model may be trained using trainingexample to identify the remedial actions, and the training examples maybe based on an analysis of the historical examples of remedial actionsand surgical outcomes.

Disclosed embodiments may include outputting a recommended remedialaction. Outputting a recommended remedial action may includetransmitting a recommendation to a device, causing a notification to bedisplayed on an interface, playing a sound, providing haptic feedback,and/or any other method of conveying a desired message, whether to anoperating room, a device associated with a surgeon (e.g., a humansurgeon and/or a surgical robot), and/or to any other system. Forexample, outputting a recommended remedial action may includetransmitting a notification to a computer, a mobile device, an externaldevice, a surgical robot, and/or any other computing device.

Further, in some embodiments a method may include updating a schedulingrecord associated with a surgical room related to a surgical procedurein response to predicted outcome dropping below a threshold. Forexample, a change in an expected duration of a surgery may trigger anautomated change in a scheduling record, such that a surgery on a nextpatient is pushed back in time to account for a delay in a currentoperation. More general, a change in any predicted outcome may beassociated with an increase or decrease in an expected duration. In someembodiments, a data structure (e.g., data structure of FIG. 17 ) maycorrelate predicted outcomes with respective expected durations ofsurgery. A model (e.g., a regression model or a trained machine learningmodel) may be used to generate an expected duration based on predictedoutcomes, consistent with the present embodiments. Thus, if a predictedoutcome change impacts a duration of surgery, a surgical schedule may beautomatically updated to inform succeeding medical staff of change inoperating room schedule. The update may be automatically displayed on anelectronic operating room scheduling board. Alternatively oradditionally, the update may be broadcast via email or other messagingapp to accounts associated with the impacted medical professionals.Scheduling may be correlated to predicted outcome as discussed, butmight also correlate to other factors. For example, even if thepredicted outcome does not change, machine vision analysis performed onthe video footage of the surgical procedure may reveal that the surgeryis behind schedule (or ahead of schedule), and an update to the schedulemay be automatically pushed, as previously discussed.

FIG. 31 is a flowchart illustrating an example process 3100 for updatinga predicted outcome during surgery, consistent with the disclosedembodiments. Process 3100 may be performed by at least one processor,such as one or more microprocessors. In some embodiments, process 3100is not necessarily limited to the steps illustrated, and any of thevarious embodiments described herein may also be included in process3100. As one of skill in the art will appreciate, steps of process 3100may be performed by a system including, for example, components ofsystem 1401. In some embodiments, a non-transitory computer readablemedium including instructions that, when executed by at least oneprocessor, cause the at least one processor to execute operations forupdating a predicted outcome according to process 3100. In someembodiments, process 3100 may be performed in real time during asurgical procedure.

At step 3102, the process may include receiving, from at least one imagesensor arranged to capture images of a surgical procedure, image dataassociated with a first event during the surgical procedure, consistentwith disclosed embodiments. An image sensor may be positioned anywherein an operating room (e.g., above a patient, within a patient), aspreviously discussed.

At step 3104, the process may include determining, based on the receivedimage data associated with the first event, a predicted outcomeassociated with the surgical procedure, as previously discussed andillustrated with examples. As discussed, for example, determining apredicted outcome may include identifying an interaction between asurgical tool and an anatomical structure and determining a predictedoutcome based on the identified interaction. Determining a predictedoutcome may be based on a skill level of a surgeon depicted in the imagedata. In some embodiments, determining a predicted outcome may be basedon a condition of an anatomical structure depicted in the image data,and may include using a machine learning model trained to determinepredicted outcomes based on historical surgical videos and informationindicating surgical outcome corresponding to the historical surgicalvideos. One example of a predicted outcome may include a likelihood ofhospital readmission. Other examples were previously provided.

At step 3106, the process may include receiving, from at least one imagesensor arranged to capture images of a surgical procedure, image dataassociated with a second event during the surgical procedure, aspreviously discussed and illuminated with examples.

At step 3108, the process may include determining, based on the receivedimage data associated with the second event, a change in the predictedoutcome, causing the predicted outcome to drop below a threshold, asalso discussed previously. For example, determining a change in apredicted outcome may be based on a time elapsed between a particularpoint in the surgical procedure and the second event. In other examples,determining a change in a predicted outcome may be based on a magnitudeof bleeding, a change of a color of at least part of an anatomicalstructure, a change of appearance of at least part of the anatomicalstructure. Determining a condition of an anatomical structure mayinclude using a machine learning model trained using training examplesto determine the condition of the anatomical structure.

At step 3110, the process may include accessing a data structure ofimage-related data based on prior surgical procedures, as discussedpreviously and as was illustrated with examples. As mentioned, a datastructure such as the one illustrated in FIG. 17 may be accessed. Thisis but one example, and many other types and forms of data structuresmay be employed consistent with the disclosed embodiments.

At step 3112, the process may include identifying, based on the accessedimage-related data, a recommended remedial action, as describedpreviously. For example, a recommended remedial action may include arecommendation to alter a surgical process, use a different surgicaltool, call-in another surgeon, revise the surgical procedure, take abreak and/or any other action that might impact the outcome of thesurgical procedure. Identifying a remedial action may include using amachine learning model trained to identify remedial actions usinghistorical examples of remedial actions and surgical outcomes.

At step 3114, the process may include outputting the recommendedremedial action, as previously described.

Disclosed systems and methods may involve analyzing current and/orhistorical surgical footage to identify features of surgery, patientconditions, and other features to detect fluid leakage. During asurgery, fluids may leak. For example, blood, bile, or other fluids mayleak from an anatomical structure. Often, a source or an extent of fluidleakage may be unknown. If left unchecked, fluid leakage can causenegative health outcomes. Therefore, aspects of the present disclosurerelate to unconventional approaches that automatically and effectivelydetermine a source and/or extent of fluid leakage during surgery.

In accordance with the present disclosure, systems, methods and computerreadable media may be provided for analysis of fluid leakage duringsurgery. Analysis may be performed in real time during an ongoingsurgical procedure. Embodiments may include providing informationrelated to the fluid leakage to a surgeon in real time. For example,analysis of fluid leakage may enable a surgeon to identify a magnitudeand/or source of a fluid leakage, thereby allowing a surgeon to performa remedial action that mitigates fluid leakage. Fluid leakage mayinclude leakage of a fluid from inside an organ or tissue to a spaceexternal to a tissue or organ (e.g., from inside to outside a bloodvessel, from inside to outside a gall bladder, etc.). Leaked fluids mayinclude blood, bile, chyme, urine, and/or any other type of body fluid.

Analysis of fluid leakage during surgery may include receiving in realtime, intracavitary video of a surgical procedure, consistent withdisclosed embodiments. An intracavitary video may be captured by animage sensor located within a patient, consistent with disclosedembodiments. For example, an image sensor located external to a patientmay collect intracavitary video (e.g., when a cavity is opened during asurgery). Receiving an intracavitary video in real time may includereceiving a video via a network or directly from an image sensor.

Consistent with the present embodiments, an intracavitary video maydepict various aspects of a surgical procedure. For example, anintracavitary video may depict a surgical robot and/or a human surgeonperforming some or all of a surgical procedure. An intracavitary videomay depict a medical instrument, an anatomical structure, a fluidleakage situation, a surgical event, and/or any other aspect of asurgical procedure, consistent with disclosed embodiments.

Analysis of fluid leakage during surgery may involve analyzing frames ofan intracavitary video to determine an abnormal fluid leakage situationin the intracavitary video, consistent with disclosed embodiments.Analyzing frames may include using any method of image analysis todetermine an abnormal fluid leakage. For example, analyzing images mayinclude analyzing difference images (e.g., images generated bysubtracting pixel data of a preceding image from pixel data of asubsequent image), using methods of homography, applying imageregistration techniques, and/or other image processing methods. Ananalysis may employ object recognition models, machine learning models,regression models, and/or any other models. Such models may be trainedto determine an abnormal fluid leakage situation using training datacomprising historical examples. For example, a machine learning modelmay be trained using training examples to detect abnormal fluid leakagesituations and/or to determine properties of abnormal fluid leakagesituations from images and/or videos, and the trained machine learningmodel may be used to analyze the intracavitary video to determine theabnormal fluid leakage situation and/or properties of the abnormal fluidleakage situation. Some non-limiting examples of such properties mayinclude a type of fluid, a magnitude of fluid leakage, a location sourceof fluid leakage, an anatomical structure related to the fluid leakage,and so forth. An example of such training example may include anintracavitary image and/or an intracavitary video, together with a labelindicating whether an abnormal fluid leakage situation is depicted inthe intracavitary image and/or in the intracavitary video, and/ortogether with a label indicating properties of an abnormal fluid leakagesituation depicted in the intracavitary image and/or in theintracavitary video.

Determining an abnormal fluid leakage situation (i.e., an abnormal fluidleakage event) may include determining various aspects of a fluidleakage, include a presence of a fluid in or on an anatomical structure,a magnitude of fluid leakage that is over a threshold (e.g., over apredetermined quantile, over a number of standard deviations), a type offluid (e.g., blood, bile, urine, chyme, and/or other type), a locationsource of fluid leakage, and/or any other feature of a fluid leakagesituation. Some fluid leakages may be normal (e.g., below a thresholdmagnitude, in a location associated with normal fluid leakage, of anormal fluid type for a particular surgical event, etc.), while othersare abnormal (e.g., above a threshold magnitude, in an undesiredlocation, not connected to a surgical event that is associated a normalfluid leakage, and/or of an abnormal fluid type).

Disclosed techniques for determining a leakage source may includeidentifying a ruptured anatomical organ, vessel, and/or other anatomicalstructure. A ruptured anatomical structure may be identified based on ananalysis of fluid leakage properties (e.g., magnitude, flow rate, flowdirection, color, or other fluid leakage properties). A rupturedanatomical structure may include any organ, a vessel (e.g., an artery),a passageway (e.g., a trachea), a tissue (e.g., a lining), and/or anyother anatomical structure. The term rupture, as used herein, may referto any break, tear, puncture, or other damage to an anatomicalstructure.

In some embodiments, an identified ruptured anatomical structure may bevisible in image frames of an intracavitary video captured by an imagesensor in an operating room (e.g., a room as depicted in FIG. 1 ).Alternatively or additionally, a ruptured anatomical structure may notbe visible in frames of an intracavitary video (e.g., it may be obscuredby other anatomical structures) and it may be identified based oninformation reflected in frames (e.g., information relating to a fluidleakage situation). Identifying a ruptured structure may involvecomparing prior frames of an intracavitary video to subsequent frames,by using a regression model, by using a machine learning model, byperforming methods of object recognition, and/or by any other method ofimage analysis. For example, a machine learning model may be trainedusing training examples to identify ruptured anatomical organs, vessels,and/or other anatomical structures from intracavitary images and/orintracavitary videos, and the trained machine learning model may be usedto analyze the intracavitary video to identify the ruptured anatomicalorgan, vessel, and/or other anatomical structure. An example of suchtraining example may include an intracavitary image and/or anintracavitary video, together with a label indicating whether a rupturedanatomical organ, vessel, and/or other anatomical structure should beidentified for the intracavitary image and/or the intracavitary video.

Embodiments may include analyzing frames of an intracavitary video toidentify a blood splash and at least one property of a blood splash. Ablood splash may refer to a presence of blood and/or a leakage of blood.Identifying a blood splash may be based on color data of anintracavitary video. In some embodiments, a property of a blood splashmay be associated with a source of a blood splash, an intensity (rate)of a blood splash, a color of a blood splash, a viscosity of a bloodsplash, and/or a volume (magnitude) of a blood splash. More generally, aproperty of a blood splash may include any characteristic of a bloodsplash. For example, a machine learning model may be trained usingtraining examples to identify blood splashes and/or to determineproperties of blood splashes from images and/or videos, and the trainedmachine learning model may be used to analyze the intracavitary video toidentify the blood splash and/or properties of the blood splash. Somenon-limiting examples of such properties of a blood splash may include asource of the blood splash, an intensity of the blood splash, a rate ofthe blood splash, a color of the blood splash, a viscosity of the bloodsplash, a volume of the blood splash, a magnitude of the blood splash,and so forth. An example of such training example may include anintracavitary image and/or an intracavitary video, together with a labelindicating whether a blood splash is depicted in the intracavitary imageand/or in the intracavitary video, and/or together with a labelindicating properties of a blood splash depicted in the intracavitaryimage and/or in the intracavitary video.

Embodiments may include analyzing frames of an intracavitary video toidentify a spray of blood and/or to identify at least one property of aspray of blood. In some examples, identifying a spray of blood may bebased on color data of an intracavitary video, on motion within theintracavitary video, and so forth. In some embodiments, a property of aspray of blood may be associated with a source of a spray of blood, anintensity (rate) of a spray of blood, a color of the sprayed blood, amotion (such as speed, direction, etc.) of the sprayed blood, and/or avolume (magnitude) of the sprayed blood. More generally, a property of aspray of blood may include any characteristic of the sprayed bloodand/or of the spray. For example, a machine learning model may betrained using training examples to identify sprays of blood and/or todetermine properties of sprays of blood from images and/or videos, andthe trained machine learning model may be used to analyze theintracavitary video to identify the spray of blood and/or properties ofthe spray of blood. An example of such training example may include anintracavitary image and/or an intracavitary video, together with a labelindicating whether a spray of blood is depicted in the intracavitaryimage and/or in the intracavitary video, and/or together with a labelindicating properties of a spray of blood depicted in the intracavitaryimage and/or in the intracavitary video.

Further, analyzing frames of an intracavitary video may includedetermining a property of an abnormal fluid leakage situation. Forexample, a property may be associated with a volume of a fluid leakage,a color of a fluid leakage, a type of fluid associated with a fluidleakage, a fluid leakage rate, a viscosity of a fluid, a reflectivity ofa fluid, and/or any other observable feature of a fluid. Further,analyzing frames may include determining a flow rate associated with afluid leakage situation, determining a volume of fluid loss associatedwith a fluid leakage situation, and/or determining any other property ofa fluid leakage situation. A property of a fluid or fluid leakagesituation may be determined based on hue, saturation, pixel values,and/or other image data. More generally, determining a property of afluid or a fluid leakage situation may include any method of imageanalysis, as disclosed herein. For example, determining a property of afluid or a fluid leakage situation may include usage of a trainedmachine learning model, as described above.

Consistent with the present embodiments, fluid leakage analysis mayinclude storing an intracavitary video, and, upon determining anabnormal leakage situation in current video, analyzing prior historicalframes of stored intracavitary video to determine a leakage source, forexample via a comparison, consistent with disclosed embodiments. Anintracavitary video may be stored in memory, in a data structure (e.g.,data structure of FIG. 17A), and so forth. For example, an abnormalleakage situation may be determined when an amount of leaked fluids isabove a selected quantity (for example, above a selected threshold usedto distinguish abnormal leakage situations from normal leakagesituations), and at that point a leakage source may not be visible inthe current video (for example, the leakage source may be covered by theleaked fluids, may be outside the current field of view of the currentvideo, and so forth). However, the leakage source may be visible in theprior historical frames of stored intracavitary video, and the priorhistorical frames of stored intracavitary video may be analyzed toidentify the leakage source, for example as described above. In anotherexample, an abnormal leakage situation may be determined by analyzingthe current video using a first algorithm, and at that point a leakagesource may not be visible in the current video. In response to suchdetermination, a second algorithm (which may be more computationallyintense or otherwise different from the first algorithm) may be used toanalyze the prior historical frames of stored intracavitary video toidentify the leakage source, which may be visible in the priorhistorical frames of stored intracavitary video. In yet another example,a trigger (such as user input, an event detect in the current video, aninput from a sensor connected to the patient undergoing the surgicalprocedure, etc.) may cause an analysis of the current video to determinethe abnormal leakage situation. Further, in some examples, in responseto such determination, the prior historical frames of storedintracavitary video may be analyzed to identify the leakage source, forexample as described above.

Analyzing prior frames to determine a leakage source may includecomparing frames at different time points (e.g., at two or more timepoints). For example, embodiments may include generating differenceimages (e.g., by subtracting pixel data of frames at two different timepoints) and analyzing the generated difference images. In anotherexample, comparing frames may involve determining a property of a fluidleakage situation at different time points and determining a change inthe property.

Embodiments may include instituting a remedial action when an abnormalfluid leakage situation is determined. A remedial action may include anynotification, suggested response, or counteractive measure associatedwith an abnormal fluid leakage situation. The remedial action may be thesame regardless of varying characteristics of a fluid leakage or mayvary based on varying characteristics of the fluid leakage. In thelatter situation, instituting a remedial action may include selecting aremedial action from various options. Thus, in the latter situation, aselection of a remedial action may depend on a determined property orcharacteristic of a fluid leakage situation. For example, if adetermined extent of bleeding is below a certain threshold and thesource of the bleeding is identified, an associated remedial action maybe a recommendation or instruction to apply pressure to the source ofthe bleeding. If a more significant rupture is detected, the remedialaction may involve a recommendation or instruction to suture the sourceof the bleeding. Depending on the type of fluid associated with theleakage, the extent of the leakage, and characteristics of the leakagesituation, many different potential remedial actions may be possible. Toassist with selecting an appropriate remedial action, a data structuremay store relationships between fluid leakage situations, remedialactions, and outcomes. Further, a statistical model may be fit based onhistorical fluid leakage situations, remedial actions, and outcomes, anda remedial action may be selected based on model output. Alternativelyor additionally, a selection may be based on output of a machinelearning model trained to select remedial actions based on historicalfluid leakage situations, remedial actions, and outcomes. In otherexamples, a data structure may store relationships between fluid leakagesituations and recommended remedial actions, and the remedial action maybe selected from the data structure based on properties and/orcharacteristics of the fluid leakage situation. Such data structure maybe based on user inputs. For example, in response to a fluid leakagesituation with an identified leakage source, a first remedial action maybe selected (such as sealing the leakage source using a surgical robot),while in response to a fluid leakage situation with no identifiedleakage source, a second remedial action may be selected (such asproviding notification to a user).

Consistent with the present embodiments, instituting a remedial actionmay include providing a notification of a leakage source. A notificationmay include a message identifying the source of the leakage, such as aruptured vessel, a ruptured organ, and/or any other ruptured anatomicalstructure. For example, a notification may include an identified leakinganatomical structure, a fluid leakage property (e.g., a volume, flowrate, fluid type, duration of a fluid leakage), and/or any otherinformation related to a fluid leakage situation. Further, anotification may include a suggested course of action that may be takento remediate or otherwise respond to the leakage. In another example, anotification may include a visual indicator of a leakage source, forexample as an overlay over an image and/or a video captured from thesurgical procedure, as an indicator in an augmented reality device, andso forth. Providing a notification may involve transmitting anotification to a device, causing a notification to be displayed at aninterface, playing a sound, providing haptic feedback, and/or any othermethod of outputting information such as disclosed above. A notificationmay be provided to a device in an operating room (e.g., as depicted inFIG. 1 ), to a device associated with a surgeon (e.g., a human surgeonand/or a surgical robot), and/or to any other system. For example,outputting a notification may include transmitting a notification to acomputer, a mobile device, an external device, a surgical robot, and/orany other computing device.

In some embodiments, a remedial action may include sending instructionsto a robot. Such instructions may direct the robot to undertake anaction that remediates or assists in remediating the leakage.Alternatively or additionally, the instruction may direct the robot tocease a current course of action and/or to move aside to permit humanintervention.

Remedial actions can be based on a variety of inputs. For example,instituting a remedial action may be based on a flow rate, a volume offluid loss, and/or any property of a fluid or fluid leakage situation,for example as described above. Remedial actions may be based onstatistical analysis of properties of fluids or fluid leakagesituations. For example, a remedial action may be selected based onknown (or determined) correlations between properties of a fluid leakagesituation and outcomes. Further, a data structure such as a datastructure of FIG. 17A may correlate properties of fluid leakagesituations with outcomes. Statistical analysis may include using aregression model to identify a remedial action (e.g., a regression modelfit to historical data that includes fluid leakage data, remedial actiondata, and outcome data).

Consistent with the present embodiments, analyzing frames ofintracavitary video to determine an abnormal fluid leakage situation inintracavitary video may include determining whether a fluid leakagesituation is abnormal. Some fluid leakage situations may be normal(e.g., below a threshold magnitude, in a location associated with normalfluid leakage, of a normal fluid type for a particular surgical event,etc.). Some fluid leakage situations may be abnormal (e.g., above athreshold magnitude, in a location associated with abnormal fluidleakage, of an abnormal fluid type for a particular surgical event,etc.). Properties of fluid leakage situations classified as normaland/or abnormal may be stored in a data structure such as one depictedin FIG. 17A. Determining whether a fluid leakage situation is abnormalmay include using a regression model (e.g., a model fit to historicalexamples) and/or a trained machine learning model (e.g., a model trainedto determine whether a determined fluid leakage situation is abnormalusing historical examples). For example, a machine learning model may betrained using training examples to determine whether fluid leakagesituations are normal or abnormal based on information related to thefluid leakage situations (such as images and/or videos of the fluidleakage situations, properties of the fluid leakage situationsdetermined by analyzing videos depicting the fluid leakage situations asdescribed above, sources of the fluid leakage situations, amounts ofleakage, types of fluid leakage situations, type of fluid,characteristics of the patient, the surgical phase that the leakagesituation occurred at, etc.), and the trained machine learning model maybe used to analyze information related to the fluid leakage situationand determining whether the fluid leakage situation is abnormal. Anexample of such training example may include information related to aparticular fluid leakage situation, together with a label indicatingwhether the particular fluid leakage situation is abnormal.

Some embodiments may include analyzing frames of an intracavitary videoto determine a property of a detected fluid leakage situation, forexample as described above. A determined property may be associated witha volume of fluid leakage, a type of fluid leakage, rate of fluidleakage, a source of fluid leakage, and/or any other observable featureof surgical procedure. The determined property may then be used, forexample, to ascertain whether the fluid leakage is normal or abnormal.Analyzing the frames to determine such a property may include any methodof image analysis as described herein.

In some embodiments, determining whether a fluid leakage situation is anabnormal fluid leakage situation may be based on a measurement of ablood pressure and/or any other vital sign of a patient undergoing asurgical procedure. The vital signs may be derived from surgical videothrough image analysis techniques described herein, and/or may bederived from sensors configured to measure vital signs. In addition, tousing vital signs as a possible indicator of an abnormality, anabnormality may be based on any characteristic of a surgical procedure,such as a surgical event, a type of surgery, and/or any other aspect ofa surgical procedure. For example, in response to a first measurement ofthe blood pressure of the patient undergoing the surgical procedure, aparticular fluid leakage situation may be determined to be normal, andin response to a second measurement of the blood pressure of the patientundergoing the surgical procedure, the particular fluid leakagesituation may be determined to be abnormal.

Disclosed embodiments may include instituting a remedial action inresponse to a determination that a detected fluid leakage situation isan abnormal fluid leakage situation. In addition, some embodiments mayinclude forgoing institution of a remedial action in response to adetermination that fluid leakage is normal. Forgoing a remedial actionmay include delaying a remedial action for a time period orindefinitely. For example, if an analysis of a leakage results in adetermination that no remedial action is necessary, remedial action maybe forgone. Or, if remedial action already began and further analysisrevealed that the remedial action is unnecessary, forgoing a remedialaction may include providing an updated notification (e.g., anotification may change a recommended remedial action or otherwisepresent information that differs from a previous notification).

FIG. 32 is a flowchart illustrating an example process 3200 for enablingfluid leak detection during surgery, consistent with the disclosedembodiments. Process 3200 may be performed by at least one processor,such as one or more microprocessors. In some embodiments, process 3200is not necessarily limited to the steps illustrated, and any of thevarious embodiments described herein may also be included in process3200. Steps of process 3200 may be performed by a system including, forexample, components of system 1401. In some embodiments, process 3200may be embodied in a non-transitory computer readable medium includinginstructions that, when executed by at least one processor, cause the atleast one processor to execute operations for analysis of fluid leakageduring surgery. In some embodiments, process 3200 may be performed inreal time during a surgical procedure.

At step 3202, the process may include receiving, receiving in real time,intracavitary video of a surgical procedure, consistent with disclosedembodiments. Receiving an intracavitary video in real time may includereceiving a video via a network or directly from an image sensor. Insome embodiments an intracavitary video may depict a surgical robotperforming some or all of a surgical procedure, as previously discussed.

At step 3204, the process may include analyzing frames of theintracavitary video to determine an abnormal fluid leakage situation inthe intracavitary video as previously discussed and illustrated withexamples. As discussed, for example, a fluid may include blood, bile,urine, and/or any other type of body fluid. Determining a leakage sourcemay include identifying a ruptured anatomical organ, identifying aruptured vessel, and/or identifying any other ruptured anatomicalstructure. In some embodiments, step 3204 may include analyzing framesof an intracavitary video to identify a blood splash and at least oneproperty of the blood splash. A property of a blood splash may beassociated with a source of the blood splash, an intensity (rate) of ablood splash, a color of a blood splash, a viscosity of a blood splash,and/or a volume (magnitude) of a blood splash. Analyzing frames of anintracavitary video may include determining a property of an abnormalfluid leakage situation. For example, a property may be associated witha volume of a fluid leakage, a color of a fluid leakage, a type of fluidassociated with a fluid leakage, a fluid leakage rate, a viscosity of afluid, a reflectivity of a fluid, and/or any other property of a fluid.Further, analyzing frames may include determining a flow rate associatedwith a fluid leakage situation, determining a volume of fluid lossassociated with a fluid leakage situation, and/or determining any otherproperty of a fluid leakage situation. A method may further comprisestoring an intracavitary video, and, upon determining the abnormalleakage situation, analyzing prior frames of the stored intracavitaryvideo to determine a leakage source.

At step 3206, the process may include instituting a remedial action whenthe abnormal fluid leakage situation is determined. A selection of aremedial action may depend on at least one property of an identifiedblood splash, for example. In some embodiments, a selection of theremedial action may depend on a determined property of a fluid leakagesituation.

Disclosed systems and methods may involve analyzing surgical footage toidentify events during the surgical procedure, which may affect the postdischarge risk for the patient. Post discharge risk for a patient mayneed to be identified after a surgical procedure, based onintraoperative events during the surgical procedure and based on patientcharacteristics. Post discharge risk may be determined by identifyingevents during the surgical procedure and using historical data todetermine how the identified events may affect the post discharge riskfor a patient. Therefore, there is a need for analyzing surgical footageand identifying events during a surgical procedure that may influencepost discharge risk for the patient.

Aspects of this disclosure may relate to predicting post discharge riskafter a surgical procedure, including methods, systems, devices, andcomputer readable media.

For ease of discussion, a method is described below, with theunderstanding that aspects of the method apply equally to systems,devices, and computer readable media. For example, some aspects of sucha method may occur electronically over a network that is either wired,wireless, or both. Other aspects of such a method may occur usingnon-electronic means. In the broadest sense, the method is not limitedto particular physical and/or electronic instrument, but rather may beaccomplished using many differing instruments.

Consistent with disclosed embodiments, a method for predicting postdischarge risk may involve accessing frames of video captured during aspecific surgical procedure on a patient. As used herein, a video mayinclude any form of recorded visual media, including recorded imagesand/or sound. For example, a video may include a sequence of one or moreimages captured by an image capture device, such as cameras 115, 121,123, and/or 125, as described above in connection with FIG. 1 . Theimages may be stored as individual files or may be stored in a combinedformat, such as a video file, which may include corresponding audiodata. In some embodiments, a video may be stored as raw data and/orimages output from an image capture device. In other embodiments, thevideo may be processed. For example, video files may include Audio VideoInterleave (AVI), Flash Video Format (FLV), QuickTime File Format (MOV),MPEG (MPG, MP4, M4P, or any other format), Windows Media Video (WMV),Material Exchange Format (MXF), or any other suitable video fileformats.

The video footage may refer to a video that has been captured by animage capture device. In some embodiments, the video footage may referto a video that includes a sequence of images in the order in which theywere originally captured. For example, video footage may include a videothat has not been edited to form a video compilation. In otherembodiments, the video footage may be edited in one or more ways, suchas to remove frames associated with inactivity during a surgicalprocedure or to otherwise compile frames not originally capturedsequentially. Accessing the video footage may include retrieving thevideo footage from a storage location, such as a memory device. Thevideo footage may be accessed from a local memory, such as a local harddrive, or may be accessed from a remote source, for example, through anetwork connection. Consistent with the present disclosure, indexing mayrefer to a process for storing data such that it may be retrieved moreefficiently and/or effectively. A process of indexing video footage mayinclude associating one or more properties or indicators with the videofootage such that the video footage may be identified based on theproperties or indicators.

A surgical procedure may include any medical procedure associated withor involving manual or operative procedures on a patient's body.Surgical procedures may include cutting, abrading, suturing, or othertechniques that involve physically changing body tissues and/or organs.Surgical procedures may also include diagnosing patients oradministering drugs to patients. Some examples of such surgicalprocedures may include a laparoscopic surgery, a thoracoscopicprocedure, a bronchoscopic procedure, a microscopic procedure, an opensurgery, a robotic surgery, an appendectomy, a carotid endarterectomy, acarpal tunnel release, a cataract surgery, a cesarean section, acholecystectomy, a colectomy (such as a partial colectomy, or a totalcolectomy), a coronary angioplasty, a coronary artery bypass, adebridement (for example of a wound, a burn, or an infection), a freeskin graft, a hemorrhoidectomy, a hip replacement, a hysterectomy, ahysteroscopy, an inguinal hernia repair, a knee arthroscopy, a kneereplacement, a mastectomy (such as a partial mastectomy, a totalmastectomy, or a modified radical mastectomy), a prostate resection, aprostate removal, a shoulder arthroscopy, a spine surgery (such as aspinal fusion, a laminectomy, a foraminotomy, a diskectomy, a diskreplacement, or an interlaminar implant), a tonsillectomy, a cochlearimplant procedure, brain tumor (for example meningioma) resection,interventional procedures such as percutaneous transluminal coronaryangioplasty, transcatheter aortic valve replacement, minimally Invasivesurgery for intracerebral hemorrhage evacuation, or any other medicalprocedure involving some form of incision. While the present disclosureis described in reference to surgical procedures, it is to be understoodthat it may also apply to other forms of medical procedures orprocedures generally.

In some exemplary embodiments, the accessed video footage may includevideo footage captured via at least one image sensor located in at leastone of a position above an operating table, in a surgical cavity of apatient, within an organ of a patient or within a vasculature of apatient. An image sensor may be any sensor capable of recording a video.An image sensor located in a position above an operating table mayinclude an image sensor placed external to a patient configured tocapture images from above the patient. For example, the image sensor mayinclude cameras 115 and/or 121, as shown in FIG. 1 . In otherembodiments, the image sensor may be placed internal to the patient,such as, for example, in a cavity. As used herein, a cavity may includeany relatively empty space within an object. Accordingly, a surgicalcavity may refer to a space within the body of a patient where asurgical procedure or operation is being performed. It is understoodthat the surgical cavity may not be completely empty but may includetissue, organs, blood, or other fluids present within the body. An organmay refer to any self-contained region or part of an organism. Someexamples of organs in a human patient may include a heart or liver. Avasculature may refer to a system or grouping of blood vessels within anorganism. An image sensor located in a surgical cavity, an organ, and/ora vasculature may include a camera included on a surgical tool insertedinto the patient.

Accessing frames of video captured during a specific surgical proceduremay include accessing at least one of the frames, metadata related bythe frames, pixel values of pixels of the frames, information based onan analysis of the frames, and so forth. For example, the frames ofvideo captured during a specific surgical procedure may be accessed by acomputerized device reading the information from a memory, for examplefor processing by at least one processing device. For example, theprocessing device may analyze the access frames using a machine-learningmethod configured to analyze various aspects of video data, for exampleas described above. For example, the machine-learning method may beconfigured to recognize events within the video frames or recognizesurgical instruments, anatomical structures and interactions betweensurgical instruments and anatomical structures by analyzing the videoframes, and so forth. In some cases, accessing frames of video mayinclude accessing the frames by a healthcare professional such as asurgeon, anesthesiologist, or any other healthcare professional. In somecases, the video frames may be accessed by a patient, a family member ofa patient, or any other authorized party.

Aspects of this disclosure may include accessing stored historical dataidentifying intraoperative events, and associated outcomes. As usedherein, an intraoperative event for the surgical procedure (alsoreferred to as a surgical event) may refer to an action that isperformed as part of a surgical procedure, such as an action performedby a surgeon, a surgical technician, a nurse, a physician's assistant,an anesthesiologist, a doctor, any other healthcare professional, asurgical robot, and so forth. The intraoperative surgical event may be aplanned event, such as an incision, administration of a drug, usage of asurgical instrument, an excision, a resection, a ligation, a graft,suturing, stitching, or any other planned event associated with asurgical procedure or phase. Additionally or alternatively, anintraoperative event may also refer to an event occurring to ananatomical structure and/or to a medical instrument related to thesurgical procedure, whether the event includes an action performed by ahealthcare professional or not. One example of such intraoperative eventmay include a change in a condition of an anatomical structure. Anotherexample of such intraoperative event may include a change in a state ofa medical instrument (for example, from ‘partly filled’ to ‘filled’).

An exemplary surgical intraoperative event for a laparoscopiccholecystectomy surgery may include trocar placement, calot's triangledissection, dipping and cutting of cystic duct and artery, gallbladderdissection, gallbladder packaging, cleaning and coagulation of liverbed, gallbladder retraction, and so forth. In another example, surgicalevents of a cataract surgery may include povidone-iodine injection,corneal incision, capsulorhexis, phaco-emulsification, corticalaspiration, intraoculariens implantation, intraocular-lens adjustment,wound sealing, and so forth. In yet another example, surgicalcharacteristic events of a pituitary surgery may include preparation,nasal incision, nose retractor installation, access to the tumor, tumorremoval, column of nose replacement, suturing, nose compressinstallation, and so forth. Some other examples of surgicalcharacteristic events may include incisions, laparoscope positioning,suturing, and so forth.

In some embodiments, the surgical intraoperative event may include anadverse event or a complication. Some examples of adverse surgicalevents may include bleeding, mesenteric emphysema, injury, conversion tounplanned open surgery (for example, abdominal wall incision), incisionsignificantly larger than planned, and so forth. Some examples ofintraoperative complications may include hypertension, hypotension,bradycardia, hypoxemia, adhesions, hernias, atypical anatomy, duraltears, periorator injury, arterial occlusions, and so forth. In somecases, surgical events may include other errors, including technicalerrors, communication errors, management errors, judgment errors,decision-making errors, errors related to medical equipment utilization,miscommunication, and so forth. In various embodiments, events may beshort or may last for a duration of time. For example, a short event(e.g., incision) may be determined to occur at a particular time duringthe surgical procedure, and an extended event (e.g., bleeding) may bedetermined to occur over a time span. In some cases, extended events mayinclude a well-defined beginning event and a well-defined ending event(e.g., beginning of suturing and ending of the suturing), with suturingbeing an extended event. In some cases, extended events are alsoreferred to as phases during a surgical procedure.

In some cases, a surgical event may identify a group of sub-events(i.e., more than one sub-event or steps). For example, an event ofadministering general anesthesia to a patient may include several stepssuch as a first step of providing a medication to a patient via an IVline to induce unconsciousness, and a second step of administering asuitable gas (e.g., isoflurane or desflurane) to maintain the generalanesthesia.

Historical data may include any information related to or based onhistorical (i.e., previously performed) surgical procedures. Forexample, the historical data may include historical surgical footage,information based on an analysis of historical surgical footage,historical notes from one or more healthcare providers, historical datafrom medical devices (e.g., historical vital signals collected during ahistorical surgical procedure), historical audio data, historical datacollected from various sensors (e.g., image sensors, chemical sensors,temperature sensors, electrical sensors), or any other historical datathat may be related to one or more historical surgical procedures.

Accessing stored historical data identifying intraoperative events andassociated outcomes may include accessing a database containinginformation about intraoperative events and associated outcomes. Forexample, a database may include a data structure, such as a table,database, or other organization of data that maintains historicalintraoperative events and historical outcomes associated withintraoperative events. For example, an intraoperative event may be“bleeding,” and a historically associated outcome may be “anemia.” Insome cases, one intraoperative event may have associated outcomes withdifferent characteristics. For example, an intraoperative event“bleeding” may have a first associated outcome “loss of hemoglobin” witha first characteristic “dropping from 16 g/dL to 13 g/dL” and a secondassociated outcome “loss of hemoglobin” with second characteristic“dropping from 15 g/dL to 12 g/dL.” In some cases, an intraoperativeevent such as “bleeding” may have a first outcome “loss of hemoglobin”and a second outcome that may be different from the first outcome (e.g.,“cardiac arrest”).

Additionally or alternatively, accessing stored historical dataidentifying intraoperative events and associated outcomes may includeaccessing the historical data by a computerized device reading at leastpart of the historical data from a memory, for example for processing byat least one processing device. The historical data may includehistorical surgical footage, information about historical intraoperativeevents, etc., and associated historical outcomes. In some cases,processing the accessed historical data may be performed by amachine-learning model configured to analyze various aspects ofhistorical surgical footage (as well as any other historical surgicaldata). For example, the machine-learning method may be configured toidentifying intraoperative events within video frames of historicalsurgical footage by recognizing surgical instruments, anatomicalstructures, and interactions between surgical instruments and anatomicalstructures in the historical surgical footage.

In some cases, accessing stored historical data identifyingintraoperative events and associated outcomes may include accessing thehistorical data by a surgeon, anesthesiologist, nurse, or any otherhealthcare professional. In some cases, the historical data may beaccessed by a patient, a family member of a patient, or any other partyauthorized to access to the historical data.

In various embodiments, a machine learning model may be used to identifyin accessed video frames of a surgical procedure at least one specificintraoperative event, for example as described above. The trainedmachine-learning model may be an image recognition model for identifyingevents. For example, the machine-learning model may analyze multiplevideo frames in order to detect a motion or other changes within theimages represented as the frames of the video. In some embodiments,image analysis may include object detection algorithms, such asViola-Jones object detection, convolutional neural networks (CNN), orany other forms of object detection algorithms. The machine-learningmodel may be configured to return a name of the event, a type of theevent, and a characteristic of the event. For example, if the event isan incision, the machine-learning model may be configured to return thename “incision” for characterizing the event, and the length and thedepth of the incision as characteristics of the event. In some cases, apredetermined list of possible names for various events may be providedto the machine-learning model, and the machine-learning model may beconfigured to select a name from the list of events for accuratecharacterization of the event.

Some aspects of disclosed embodiments may include using amachine-learning module to identify an occurrence of at least onespecific intraoperative event. For example, the machine-learning modelmay identify that a specific intraoperative event occurs by identifyinga beginning of the event. In some cases, the machine-learning model mayidentify characteristics of the event and/or an end of the specificintraoperative event.

The machine-learning model may be trained to identify intraoperativeevents using example training data. The example training input data mayinclude a historical footage of surgical procedures that includeintraoperative events. In various cases, multiple samples of trainingdata may be generated and used for training the machine-learning model.During a training process, training data (e.g., a first training data)may be used as an input data for the model, and the model may performcomputations and output an event identifying string for theintraoperative event (e.g., a name of the event). In variousembodiments, the event identifying string may be compared with a knownhistorical name of a corresponding intraoperative event to evaluate anassociated error for the model. If the error is below a predeterminedthreshold value, the model may be trained using other input data.Alternatively, if the error is above the threshold value, modelparameters may be modified, and a training step may be repeated usingthe first training data.

In addition to or as an alternative to detection with a machine-learningmodel the characteristic event may be detected in the video framesreceived from image sensors using various other approaches. In oneembodiment, the characteristic event may be identified by a medicalprofessional (e.g., a surgeon) during the surgical procedure. Forexample, surgeon may identify the characteristic event using a visual oran audio signal from the surgeon (e.g., a hand gesture, a body gesture,a visual signal produced by a light source generated by a medicalinstrument, a spoken word, or any other suitable signal) that may becaptured by one or more image sensors/audio sensors and recognized as atrigger for the characteristic event.

Aspects of this disclosure may also include identifying at least onespecific intraoperative event based on at least one of a detectedsurgical tool in the accessed frames, a detected anatomical structure inthe accessed frames, an interaction in the accessed frames between asurgical tool and an anatomical structure, or a detected abnormal fluidleakage situation in the accessed frames.

A surgical tool may be any instrument or device that may be used duringa surgical procedure, which may include, but is not limited to, cuttinginstruments (such as scalpels, scissors, or saws), grasping and/orholding instruments (such as Billroth's clamps, hemostatic “mosquito”forceps, atraumatic hemostatic forceps, Deschamp's needle, or Hopfner'shemostatic forceps), retractors (such as Farabefs Cshaped laminar hook,blunt-toothed hook, sharp-toothed hook, grooved probe, or tamp forceps),tissue unifying instruments and/or materials (such as needle holders,surgical needles, staplers, clips, adhesive tapes, or a mesh),protective equipment (such as facial and/or respiratory protectiveequipment, headwear, footwear, or gloves), laparoscopes, endoscopes,patient monitoring devices, and so forth. A surgical tool (also referredto as a medical tool or medical instrument) may include any apparatus ora piece of equipment used as part of a medical procedure.

The surgical tool may be detected in the surgical video footage usingany suitable means, for example as described above.

Similarly, to detecting the surgical tool, an anatomical structure maybe detected in the surgical footage using a machine-learning model. Ananatomical structure may be any particular part of a living organism,including, for example, organs, tissues, ducts, arteries, or any otheranatomical parts. In some cases, prosthetics, implants, or artificialorgans may be considered anatomical structures.

Detecting a surgical tool and/or an anatomical structure using a machinelearning method may be one possible approach, for example as describedabove. Additionally or alternatively, the surgical tool (or anatomicalstructure) may be detected in the surgical video footage received fromimage sensors using various other approaches. In one embodiment, thesurgical tool (or anatomical structure) may be identified by a medicalprofessional (e.g., a surgeon) during the surgical procedure. Forexample, surgeon may identify the surgical tool (or anatomicalstructure) by saying the name of the surgical tool and/or anatomicalstructure, such that the audio sound from the surgeon may be captured byone or more audio sensors and recognized by a speech recognitioncomputer-based model (or by a human operator for recording informationduring the surgical procedure).

Some aspects of the present disclosure may involve analyzing frames ofthe surgical footage to identify an interaction between the medical tooland an anatomical structure, for example as described above. Forexample, at least some of the frames of the surgical footage mayindicate a portion of the surgical footage in which a surgical operationis being performed on the anatomical structure. As discussed above, theinteraction may include any action by the medical instrument that mayinfluence the anatomical structure or vice versa. For example, theinteraction may include a contact between the medical instrument and theanatomical structure, an action by the medical instrument on theanatomical structure (such as cutting, clamping, grasping, applyingpressure, or scraping), a physiological response by the anatomicalstructure, the surgical tool emitting light towards the anatomicalstructure (e.g., surgical tool may be a laser that emits light towardsthe anatomical structure) a sound emitted towards anatomical structure,an electromagnetic field created in a proximity of the anatomicalstructure, a current induced into an anatomical structure, or any othersuitable forms of interaction.

In some cases, identifying interaction may include identifying theproximity of the surgical tool to an anatomical structure. For example,by analyzing the surgical video footage of an example surgicalprocedure, the image recognition model may be configured to determine adistance between the surgical tool and a point (or a set of points) ofan anatomical structure.

Aspects of the present disclosure may also involve detecting an abnormalfluid leakage situation in the accessed frames, for example as describedabove. The abnormal fluid leakage may include bleeding, urine leakage,bile leakage, lymph leakage, or any other leakage. The abnormal fluidleakage may be detected by a corresponding machine-learning modeltrained to detect abnormal fluid leakage events within the surgicalvideo footage captured during a surgical procedure. It should be notedthat a machine-learning model (e.g., a first machine-learning model) fordetecting surgical instruments may be configured and/or traineddifferently from a machine-learning model for detecting anatomicalstructures (e.g., a second machine-learning model), and may beconfigured and/or trained differently from a machine-learning model fordetecting abnormal leakages (e.g., a third machine-learning model).Further, the second machine-learning model may be configured and/ortrained differently than the third machine-learning model. In variousembodiments, configuring a machine-learning model may includeconfiguring (e.g., selecting) any suitable parameters of themachine-learning model. For example, if the machine-learning model is aneural network, configuring the neural network may include selecting anumber of layers for the neural network, a number of nodes for eachlayer, weights of the neural network, or any other suitable parametersof the neural network.

Aspects of disclosed embodiments may include analyzing accessed framesand, based on information obtained from historical data, identifying inthe accessed frames at least one specific intraoperative event. Aspreviously described, and consistent with various embodiments, a processof analyzing the accessed frames may be performed by a suitablemachine-learning model such as an image recognition algorithm, asdescribed above, consistent with disclosed embodiments. In variousembodiments, information obtained from historical data may be used totrain the image recognition algorithm to recognize specificintraoperative events base on accessed frames of surgical footage, aspreviously described. In one example, the historical data may include astatistical model and/or a machine learning model based on an analysisof information and/or video footage from historical surgical procedures(for example as described above), and the statistical model and/or themachine learning model may be used to analyze the accessed frames andidentify in the accessed frames the at least one specific intraoperativeevent.

Aspects of this disclosure may include determining, based on informationobtained from historical data, and an identified at least oneintraoperative event, a predicted outcome associated with a specificsurgical procedure. For example, a data structure may include historicaldata representing relationships between intraoperative events andpredicted outcomes. Such data structures may be used to obtain apredicted outcome associated with a specific surgical procedure. Forexample, FIG. 32A shows an example graph 3200 of intraoperative eventsE1-E3 connected to possible outcomes C1-C3 using connections n11-n32. Inan example embodiment, connection n11 may include information indicatinga probability of an outcome C1 (i.e., information indicating how oftenoutcome C1 happens in surgical procedures that includes event E1). Forexample, connection n11 may indicate that given an occurrence ofintraoperative event E1, outcome C1 may happen 30 percent of the time,connection n12 may indicate that outcome C2 may happen 50 percent of thetime, and connection n13 may indicate that outcome C3 may happen 20percent of the time. Similarly, connection n22 may indicate aprobability of outcome C2, given an occurrence of intraoperative eventE2, and connection n23 may indicate a probability of outcome C3, givenan occurrence of intraoperative event E2. A connection n32 may indicatea probability of outcome C2, given an occurrence of intraoperative eventE3. Thus, once an intraoperative event is known, using informationobtained from historical data (e.g., using information from graph C100),a most probable outcome (e.g., outcome C2) may be determined based onprobability assigned to connections n11-n13. In another example, thehistorical information may include a hypergraph, a hyperedge of thehypergraph may connect a plurality of intraoperative events with anoutcome and may indicate a particular probability of the outcome insurgical procedures that included the plurality of events. Thus, once aplurality of intraoperative events is known, using information obtainedfrom historical data (e.g., from the hypergraph), a most probableoutcome may be determined based on probability assigned to thehyperedges. In some examples, probabilities assigned to edges of graphC100 or to the hyperedges of the hypergraph may be based on an analysisof historical surgical procedures, for example by calculating thestatistical probability of an outcome in a group of historical surgicalprocedures that include particular group of intraoperative eventscorresponding to a particular edge or a particular hyperedge. In someother examples, the historical information may include a trained machinelearning model for predicting outcome based on intraoperative events,and the trained machine learning model may be used to predict theoutcome associated with the specific surgical procedure based on theidentified at least one intraoperative event. In one example, thetrained machine learning model may be obtained by training a machinelearning algorithm using training examples, and the training examplesmay be based on historical surgical procedure. An example of suchtraining example may include a list of intraoperative surgical events,together with a label indicating an outcome corresponding to the list ofintraoperative surgical events. In one example, two training examplesmay have the same list of intraoperative surgical events, while havingdifferent label indicating different outcomes.

In some cases, a predicted outcome may be a particular predicted eventfollowing a surgical procedure. For example, the predicted outcome maybe a post-discharge mishap, a post-discharge adverse event, apost-discharge complication, or an estimate of a risk of readmission. Insome cases, the predicted outcome may be a set of events. For example,such a set of events may include events in which the “well-being” of apatient is evaluated. These events, when the “well-being” of the patientis evaluated, may occur at specific points in time (e.g., at specifichours during the day following the surgical procedure, at specific daysfollowing the surgical procedure, at specific weeks, month, years, orother time intervals, following the surgical procedure). The“well-being” may be evaluated using any suitable objective measure suchas, e.g., using imaging such as a CAT scan, ultrasound imaging, visualinspection, presence of complications that can be determined during aphysical exam, or any other suitable way or test for evaluating awell-being of a patient (e.g., via a blood test). Well-being may also bedetermined from subjective measures such as by asking the patient todescribe his/her overall condition.

The determining of the predicted outcome associated with the surgicalprocedure based on the determined intraoperative event may beaccomplished using a statistical analysis. For example, historicalsurgical data for past (also referred to as historical) surgicalprocedures containing an intraoperative event, may be analyzed todetermine a historical outcome for such past surgical procedures. Forexample, for a given type of a historical surgical procedure, surgicaloutcome statistics may be collected, as shown in FIG. 32B. For instance,a probability distribution 3201A represented by bars 3211A-3217A (hereinalso referred to as probability bars) may determine a probability ofcorresponding outcomes C1-C4, when an intraoperative event is notpresent (e.g., when an adverse intraoperative event such as bleeding,cardiac arrest, or any other adverse event is not present). Similarly,probability distribution 3201B represented by probability bars3211B-3217B may determine a probability of corresponding outcomes C1-C4when the intraoperative event (e.g., an adverse intraoperative event) ispresent. In an example embodiment, outcome C1 may correspond to aspecific post-discharge mishap (e.g., a foreign object such as gauze isleft in a patient's body), outcome C2 may correspond to a specificpost-discharge adverse event (e.g., bleeding, pain, nausea, confusion,or any other adverse event), outcome C3 may correspond to apost-discharge complication (e.g., paralysis, pain, bleeding, or anyother complication), and outcome C4 may correspond to an elevated riskof readmission. It should be noted that any other suitable outcomes maybe used to evaluate the surgical procedure (e.g., an outcome thatevaluates an objective measure of a patient's “well-being” several daysafter the surgical procedure). In an example embodiment, the height ofprobability bars 3211A-3217A and 3211B-3217B may relate to a probabilityof occurrence of corresponding outcomes C1-C4.

In an example embodiment, an intraoperative event may affect theprobabilities of occurrence of outcomes C1-C4, as shown by bars3211B-3217B that have different heights than corresponding bars3211A-3217A. In an illustrative example, if the intraoperative eventcorresponds to a cardiac arrest during a surgical procedure, bar 3213Bcorresponding to a probability of outcome C2 (e.g., confusion) may behigher than bar 3211B corresponding to a probability of outcome C2 whenthe intraoperative event was not detected during the surgical procedure.

In some cases, a statistical analysis may be used to determine thepredicted outcome associated with the surgical procedure based on adetermination of several intraoperative events that may occur during thesurgical procedure. For example, FIG. 33 shows a probabilitydistribution 3201A with probability bars 3211A-3217A corresponding toprobability for outcomes C1-C4 when there are no adverse intraoperativeevents present (as described above). FIG. 33 also show a probabilitydistribution 3201B with probability bars 3211B-3217B corresponding toprobability for outcomes C1-C4 when there is a first adverse eventlabeled “B” present during a surgical procedure. Likewise, FIG. 33 alsoshows a probability distribution 3201C with probability bars 3211C-3217Ccorresponding to probability for outcomes C1-C4 when there is a secondadverse event labeled “C” present during a surgical procedure. Further,using statistical data for surgical procedures that include event “B”and event “C”, with event “B” starting prior to a start of event “C”,the probability distribution 3201BC may be determined as shown by bars3211BC-3217BC corresponding to probability for outcomes C1-C4.

Additionally or alternatively, using statistical data for surgicalprocedures that include event “B” and event “C”, with event “B” startingafter the start of event “C”, the probability distribution 3201CB may bedetermined as shown by bars 3211CB-3217CB corresponding to probabilityfor outcomes C1-C4. It should be noted that other probabilitydistributions (besides distributions 3201B, 3201C, 3201BC, and 3201CB)may be determined using a suitable statistical data depending on variouscharacteristics of events “B”, and/or “C” and/or combination of thereof.For instance, an event characteristic may include a duration of time forthe event, a starting time for the event, a finishing time for theevent, or any other suitable characteristic (e.g., if an event is anincision, an event characteristics may be a length of the incision; ifthe event is a cardiac arrest, the event characteristics may be bloodpressure values during the cardiac arrest; or any other suitablecharacteristic). An example embodiment of how the probabilitydistribution is affected by an event characteristic is shown in FIG. 34by plotting heights of bars 3411-3417 corresponding to probability foroutcomes C1-C4 in a three-dimensional Cartesian system. As shown in FIG.34 , one axis is a probability for outcomes C1-C4, another axis denotesthe outcome (e.g., outcomes C1-C4), and the third axis denotes “EventCharacteristics” of an intraoperative event and is represented by anumerical value (herein referred to as the event characteristic value)such as, for example, incision length for intraoperative event being anincision. FIG. 34 shows that bar heights for bars 3411-3417 may changecontinuously as the event characteristic value changes, while in otherexamples the event characteristic value may be discrete. For a givenevent characteristic value (e.g., V1, as shown in FIG. 34 ) the heightvalue (e.g., H1) for an example bar (e.g., 3415), corresponding to aprobability of outcome C3 in case of an event characteristic V1, may beinterpolated using nearby height values for bar 3415, when height valueH1 is not known for value V1.

Aspects of this disclosure may include determining a predicted outcomebased on at least one of a characteristic of the patient, an electronicmedical record, or a postoperative surgical report. The patientcharacteristic may include an age, gender, weight, height, and/or anyother information directly or indirectly characterizing the patient(e.g., whether the patient has relatives who can care for the patientmay be a characteristic that indirectly influences the predicted outcomeof the surgical procedure), to the extent such characteristics mayinfluence the predicted outcome of the surgical procedure. Some othernon-limiting examples of patient characteristics are described above. Inone example, a similarity measure may be used to identify surgicalprocedures (for example in the historical data) that are similar to thespecific surgical procedure (for example using a k-Nearest Neighborsalgorithm, using an exhaustive search algorithm, etc.), and theidentified similar surgical procedures may be used to determine thepredicted outcome, for example by calculating a statistical function ofthe outcomes of the identified similar surgical procedures (such asmean, median, mode, and so forth). The similarity measure may be basedon at least one of a characteristic of the patient, an electronicmedical record, a postoperative surgical report, intraoperative eventsthat occurred in the surgical procedures, durations of phases of thesurgical procedures, and so forth.

The electronic medical record (or any other suitable medical record,such as a paper medical record) may contain medical information for thepatient (e.g., previous surgical procedures for the patient, previous orcurrent diseases for the patient, allergies for the patient, diseases ofpatient's parents, diseases of patient's siblings, patient's mentalhealth, or any other medical information about a patient). The medicalinformation may be organized in any suitable manner (e.g., theinformation may be organized using tables, linked lists, or any othersuitable data structure). In some cases, other (non-medical) informationrelated to a patient (e.g., patient's location, diet, religion, race,occupation, fitness record, a marital status, an alcohol or tobacco use,or a previous drug use) may be contained (recorded) in the electronicmedical record. In various embodiments, such information in theelectronic medical record may be used to determine a predicted outcomefor the surgical procedure.

In various embodiments, a postoperative surgical report may further beused to determine the outcome of the surgical procedure. Thepostoperative surgical report may include any suitable informationrelated to the surgical procedure. For example, the report may include aname of the surgical procedure, patient characteristics, as discussedabove, patient's medical history, including the medical report for thepatient, other information related to the patient, as discussed above,information about surgical events that happened during the surgicalprocedure, including information about procedural surgical events, suchas actions taken during the surgical procedure, as well as informationabout adverse or positive surgical events. In an example embodiment,surgical events may include actions such as incisions, suturing, or anyother activities, performed by the surgeon. Adverse surgical events mayinclude any events that may negatively influence a surgery and apredicted outcome for a surgical procedure, such as bleeding, rupture oftissues, blood clots, cardiac arrest, or any other adverse surgicaleventualities. Positive surgical events may include determining that atleast some steps of the surgical procedure may not be necessary. Forexample, if during a surgical procedure, it is determined that thepatient does not have a tumor, removal of the tumor may not benecessary. In various embodiments, the information in the postoperativesurgical report may include surgical footage depicting the surgicalprocedure, audio data, text data, or any other suitable data recordedbefore, during, or after the surgical procedure.

In various embodiments, determining the predicted outcome based on atleast one of a characteristic of a patient, an electronic medicalrecord, or a postoperative surgical report may be achieved by analyzinghistorical surgical outcomes based on variety of parameters such aspatient characteristics, medical history data found in the electronicmedical record or various events and event characteristics described inthe postoperative surgical report.

In various embodiments, determining the predicted outcome may includeusing a machine learning model (herein also referred to as anevent-based machine-learning model) trained to determine predictedoutcomes associated with a specific surgical procedure based onintraoperative events. Additionally, the event-based machine-learningmethod may be trained to predict surgical outcomes bases on a variety ofother parameters (besides intraoperative events) such as patientcharacteristics, medical history of the patient, characteristics of oneor more healthcare professionals administering the surgical procedure orany other suitable parameters.

FIG. 35A shows an example event-based machine learning model 3513 thattakes input 3510 and outputs a predicted outcome for a surgicalprocedure 3515. Input 3510 may include input parameters 3523, as shownin FIG. 35B, such as patient characteristics and information from amedical record as previously discussed. Further, input 3510 may includeinformation from the postoperative surgical report that may includeevent data 3521, as shown in FIG. 35B. In an example embodiment, eventdata 3521 may include a list of events (e.g., events E1-EN), andsurgical footage segments V1-VN corresponding to events E1-EN. Further,data 3521 in FIG. 35B may include event starting times T1A-TNA andfinishing times T1B-TNB. Surgical footage (e.g., V1) may be a set offrames of a surgical procedure corresponding to an event (e.g., E1). Inan example embodiment, for an example surgical procedure, event E1 maybe a short event (e.g., incision) for which T1A and T1B may be about thesame time; event E2 may be an extended time (e.g., suturing) for whichT2A is the time at which the suturing started and T2B is the time atwhich the suturing ended; and event EN may be a process of administeringmedications to reverse anesthesia having corresponding starting time TNAand finishing time TNB.

In various embodiments, the event-based machine learning method may betrained using training examples, for example as described above. Forexample, a training example may be based on historical data. In anotherexample, a training example may include information related to asurgical procedure (for example as described above), together with alabel indicating an outcome.

Aspects disclosed embodiments may include determining a predictedoutcome using the trained machine-learning model to predict surgicaloutcomes based on the identified intraoperative event and an identifiedcharacteristic of a patient. For instance, an intraoperative event maybe “incision,” and one of the identified characteristics of a patientmay be “65-year-old female.” In an example embodiment, identifiedintraoperative events and one or more identified characteristics of apatient may be used as input data for the trained machine-learningmodel. For example, input data may be input 3510, as shown in FIG. 35B.In some cases, besides identified intraoperative events and anidentified characteristic of a patient, additional input data for thetrained machine-learning model may include healthcare professional data,patient medical data, and any other suitable data that has an influenceon an outcome of the surgical procedure. The event-based model may betrained, for example as described above, using various training datathat may include or be based on historical events. As described above,disclosed embodiments may include using a trained machine-learning modelto predict surgical outcomes. In various embodiments, the predictedsurgical outcome may be a distribution of probabilities for a differentset of outcomes, as shown, for example, by a plot 3201A, as shown inFIG. 33 .

Aspects of embodiments for predicting post discharge risk may alsoinclude identifying a characteristic of a patient and determining apredicted outcome associated with the surgical procedure based on theidentified patient characteristic. The predicted outcome associated withthe surgical procedure based on the identified patient characteristicmay be determined using a suitable machine-learning model, such as, forexample, model 3513, as shown in FIG. 35A.

In some embodiments, an identified patient characteristic may be basedon pre-operative patient data (e.g., the pre-operative blood testvalues, pre-operative vital signals, or any other pre-operativecharacteristics). Additionally or alternatively, an identified patientcharacteristic may be based on post-operative patient data (e.g., thepost-operative blood test values, post-operative vital signals,post-operative weight, or any other post-operative characteristics).

In various embodiments, identifying the patient characteristic mayinclude using a machine learning model to analyze frames of surgicalfootage. An example machine-learning model may be an image recognitionalgorithm, as previously described, for recognizing features withinframes of the surgical video footage captured during the surgicalprocedure. For example, the image recognition algorithm may be used torecognize features such as the size of anatomical structures that arebeing operated upon, the size of a patient, the estimated age of thepatient, gender of the patient, a race of the patient, or any othercharacteristics related to the patient. The machine-learning model foridentifying the patient characteristic may be trained, for example asdescribed above, to identify patient characteristics using trainingexamples of historical surgical procedures (including a relatedhistorical surgical video footage) and corresponding historical patientcharacteristics. The training of a machine-learning method foridentifying the patient characteristic may use any suitable approaches,as described above. In various embodiments, training themachine-learning method may use training examples based on historicalsurgical footage with labels based on output one or more patientcharacteristics corresponding to the historical surgical footage.

Additionally or alternatively, identifying patient characteristics maybe derived from an electronic medical record. For example, theelectronic medical record may be read (or parsed) using a suitablecomputer-based software application, and patient characteristics may beidentified from the read (parsed) data. For example, if the electronicrecord includes “James is a 65-year-old Caucasian male with lungdisease.” the computer-based software application may identify patientcharacteristics represented by example records, such as “Age: 65,”“Name: James” “Gender: Male,” “Medical Condition: Lung Disease,” and/or“Race: Caucasian.”

Aspects of embodiments for predicting post discharge risk may alsoinclude receiving information identifying a realized surgical outcomefollowing the surgical procedure (herein also referred to postoperativeinformation) and updating the machine-learning model by training themachine-learning model using the received information. For example, anonline machine learning algorithm and/or a reinforcement machinelearning algorithm may be used to update the machine learning modelbased on the received information. In an example embodiment, receivingthe postoperative information may include receiving visual or audio dataduring a physical examination following the surgical procedure,receiving lab results (e.g., blood test results, urine results, medicalimaging data, or any other suitable tests) following the surgicalprocedure, receiving data related to patient's vital signs (e.g., apulse of a patient, a blood pressure of the patient, or any other vitalsigns), and/or receiving notes from a healthcare provider (e.g., adoctor conducting a physical examination of the patient). In some cases,the received postoperative information may be used to determine therealized surgical outcome. For example, the received information may beanalyzed by a healthcare provider (e.g., a doctor), and the doctor mayidentify the realized surgical outcome (e.g., the realized surgicaloutcome may include a determination by the doctor that the patient doesnot require any more medical intervention). Alternatively, the receivedpostoperative information may be used by a suitable outcome-determiningmachine-learning model to determine the realized surgical outcome. Invarious embodiments, an outcome-determining machine-learning model thattakes as input postoperative information may be different than amachine-learning model for predicting an outcome of the surgery based oninformation obtained during a surgical procedure, and other relatedinformation such as patient characteristics, healthcare providercharacteristics, or a medical history of the patient, as describedabove.

In various embodiments, the received information may be used todetermine the realized surgical outcome, and the machine-learning modelfor predicting surgical outcomes based on an identified intraoperativeevent and an identified patient characteristic may be updated bytraining the machine-learning method using the received informationidentifying a realized surgical outcome. The training of amachine-learning method may be used any suitable approaches as describedabove.

In some embodiments, an output of the machine-learning model forpredicting surgical outcomes may be a probability of a predictedoutcome. In some cases, the model may be trained by comparing theprobability that is output by the model with a corresponding probabilityof a predicted outcome as inferred from historical surgical data (e.g.,historical surgical footage data) of historical surgical procedures. Forexample, using various historical surgical data, a historicalprobability of a given outcome of a surgical procedure for a given typeof a surgical procedure may be obtained. In some cases, a historicaldeviation (i.e., a deviation between the historical surgical procedureand a recommended sequence of events for the surgical procedure) may beused to determine how the historical deviation affects changes in thehistorical probability of a given outcome.

The historical probability value may be compared with a probabilityvalue returned by the machine-learning method to determine an error ofthe method. In various embodiments, if the predicted outcome returned bythe machine-learning method is a probability or a probability vector,then the suitable measure may be a difference between the probabilityand the historical probability or a difference between the probabilityvector and the historical probability vector. For instance, theprobability vector may be used to determine the probability of apredicted outcome for a set of events. For instance, if a set ofpredicted outcomes includes example outcomes C1-C4, such as “C1: beingparalyzed,” “C2: being deceased within three months,” “C3: in need ofblood transfusion within two weeks,” “C4: requiring no medicalintervention,” a probability vector for an outcome vector {C1,C2,C3,C4}may be {p1,p2,p3,p4} with p1-p4 indicating probability of outcomesC1-C4.

Aspects of embodiments for predicting post discharge risk may alsoinclude outputting a predicted outcome in a manner associating thepredicted outcome with the patient. In some embodiments, a process ofoutputting may include transmitting the predicted outcome to a receivingparty. The receiving party may include one or more healthcareprofessionals, a patient, a family member, or any other person,organization, or data storage. A process of transmitting may includetransmitting the information using any suitable electronic approach(e.g., using wired or wireless communication, as described above) to anysuitable electronic device. For example, transmitting the predictedoutcome may include transmitting the predicted outcome to adata-receiving device (e.g., a laptop, a smartphone, or any othersuitable electronic device) associated with a health care provider. Insome cases, transmitting the information may involve mailing (ordelivering in person) a physical copy (e.g., a paper copy, a CD-ROM, ahard drive, a DVD, a USB drive, or any other electronic storage device)of documents detailing the predicted outcome. Additionally oralternatively, transmitting the predicted outcome may includetransmitting the predicted outcome to at least one of a health insuranceprovider or a medical malpractice carrier.

In various embodiments, outputting the predicted outcome may be done ina manner that associates the predicted outcome with a patient. Forexample, a patient's name and/or any other suitable information aboutthe patient may be listed in a document describing a predicted outcome.

In some embodiments, transmitting the predicted outcome may includeupdating an electronic medical record associated with the patient. Aprocess of updating the electronic medical record may include replacingor modifying any appropriate data in the electronic medical record. Forexample, updating a medical record may include changing the predictedoutcome from “expected to use hands and feet after two weeks of physicaltherapy” to “expected to be paralyzed for the rest of patient's life.”

Aspects of this disclosure may include accessing a data structurecontaining recommended sequences of surgical events, and identifying atleast one specific intraoperative event based on an identification of adeviation between a recommended sequence of events for the surgicalprocedure identified in the data structure, and an actual sequence ofevents detected in the accessed frames. A process of accessing a datastructure may be performed by any suitable algorithm and/or amachine-learning model configured to identify the deviation, asdiscussed in this disclosure. For example, the machine-learning modelmay be used to access the data structure and output deviations betweenrecommended sequences of surgical events and actual events performedduring the surgery. For instance, if during an actual incision event,the incision length is shorter than an incision described by acorresponding recommended event, such deviation may be identified by themachine-learning method. The data structure containing a recommendedsequence of surgical events may be any suitable data structure describedherein, consistent with disclosed embodiments. For example, the datastructure may be a relational database having one or more databasetables. The data structure may contain a recommended sequence of eventsand may include names of the events, images corresponding to the events,video data related to the events, or any other suitable data that maydescribe the events. The data structure may define a recommendedsequence of the events by assigning to each event a number associatedwith an order of the event in the sequence.

In various embodiments, identifying a deviation between a recommendedsequence of events for the surgical procedure and an actual sequence ofevents may include various approaches discussed in this disclosure, forexample, as described in connection with FIGS. 26-28 , and relateddescription for these figures. In an example embodiment, the events ofthe surgical procedure may be identified by analyzing the surgicalfootage of the surgical procedure as previously described. Identifying adeviation between the surgical procedure and the recommended sequence ofevents may include utilizing a machine-learning approach, as describedin this disclosure. Identifying at least one specific intraoperativeevent based on an identification of the deviation may includeidentifying at least one actual event detected in the accessed framesthat deviates from the recommended sequence of events for the surgicalprocedure, and such identification may be performed by themachine-learning method for identifying the deviation.

The deviation-based model may be trained using various training datathat include historical deviations. In an example embodiment,ahistorical deviation may be determined by evaluating a deviation for ahistorical sequence of events of an example historical surgicalprocedure of a given type (e.g., bronchoscopy) and a correspondingrecommended sequence of events for the surgical procedure of the sametype. The deviation-based model may be trained using any suitabletraining process, for example as described above.

In various embodiments, identifying the deviation includes using amachine learning model trained to identify deviations from recommendedsequences of events based on historical surgical video footage,historical recommended sequences of events, and information identifyingdeviations from the historical recommended sequences of events in thehistorical video footage. Using the machine-learning method foridentifying the deviation based on historical surgical video footage,and historical recommended sequences of events are described herein andare not repeated in the interest of brevity.

In various embodiments, identifying the deviation includes comparingframes of a surgical procedure (e.g., frames accessed by any suitablecomputer-based software application or a healthcare professional foranalyzing information within the frames, as discussed above) toreference frames depicting the recommended sequence of events. Aspreviously described, the reference frames may be historical framescaptured during historical surgical procedures. In an exampleembodiment, the video frames and the reference frames depicting themandatory sequence of events may be synchronized by an event (hereinalso referred to as a starting event) that may be the same (orsubstantially similar) to a corresponding starting event of themandatory (or recommended) sequence of events. In some cases, a framedepicting the beginning of the starting event may be synchronized with areference frame depicting the starting event of the mandatory(recommended) sequence of events. In some cases, events of the surgicalprocedure may be first correlated to corresponding reference events ofthe mandatory sequence, using any suitable approaches described above(e.g., using an image recognition algorithm for recognizing events).After correlating a surgical event with corresponding reference eventsof the mandatory sequence, a frame depicting the start of the surgicalevent may be synchronized with a reference frame depicting the start ofthe corresponding mandatory event.

In various embodiments, identifying a deviation between the specificsurgical procedure and the recommended sequence of events for thesurgical procedure may be based on at least one of a detected surgicaltool in accessed frames of surgical footage, a detected anatomicalstructure in the accessed frames, or an interaction between the detectedsurgical tool and the detected anatomical structure. In some cases,identifying the deviation may be based on a detected abnormal fluidleakage situation in the surgical video footage.

For example, if it is determined (e.g., using a machine-learning method,using a visual object detector, using an indication from a healthcareprofessional, and so forth) that the surgical tool is present in aparticular anatomical region, the method may determine that deviationhas occurred. In some cases, if the surgical tool is present (asidentified in the surgical footage) in a particular anatomical regionduring a time (or a time interval) of the surgical procedure when itshould not be present, the method may determine that the deviation hasoccurred. Alternatively, in some cases, identifying the deviation mayinclude determining that a surgical tool is not in a particularanatomical region. For example, if during a time (or a time interval) ofthe surgical procedure, the surgical tool is not present in a particularanatomical region, the method may be configured to determine that thedeviation has occurred.

In some cases, when it is determined (e.g., using a machine-learningmethod, using a visual object detector, using an indication from ahealthcare professional, and so forth) that the anatomical structure ispresent in the surgical footage, it may further be determined that adeviation has occurred. For instance, if the anatomical structure isidentified in the surgical footage during a time (or a time interval) ofthe surgical procedure when it should not be present, the method maydetermine that the deviation has occurred. Alternatively, in some cases,identifying the deviation may include determining that the anatomicalstructure is not present in the surgical footage. For example, if duringa time (or a time interval) of the surgical procedure, the anatomicalstructure is not present in the surgical footage, the method may beconfigured to determine that the deviation has occurred.

Additionally or alternatively, identifying the deviation may includeidentifying an interaction between a surgical tool and an anatomicalstructure. A process of identifying the interaction between a surgicaltool and an anatomical structure may involve analyzing frames of thesurgical procedure to identify the interaction, as described above.

In various embodiments, if the interaction between a surgical tool andan anatomical structure during a surgical procedure is identified and nosuch interaction is recommended (or expected) for a reference surgicalprocedure (i.e., the surgical procedure that follows a mandatory (orrecommended) sequence of events), then the method may be configured todetermine that the deviation has occurred. Alternatively, if theinteraction between a surgical tool and an anatomical structure is notidentified (e.g., if the interaction is not present during a surgicalprocedure), and the interaction is recommended for a reference surgicalprocedure, then the method may be configured to determine that thedeviation has occurred. The method may be configured to determine thatthere is no substantial deviation of a surgical procedure and areference surgical procedure if an interaction between a surgical tooland an anatomical structure is present (or absent) in both the surgicalprocedure and the reference surgical procedure.

Aspects of the present disclosure may also involve identifying thedeviation based on a detected abnormal fluid leakage situation in thesurgical video footage. As described above, the abnormal fluid leakagemay include bleeding, urine leakage, bile leakage, lymph leakage, or anyother leakage, and may be detected (for example by a correspondingmachine-learning model) as described above. For example, if the abnormalfluid leakage is detected (as identified in the surgical footage) in aparticular anatomical region during a time (or a time interval) of thesurgical procedure when it should not be present, the method maydetermine that the deviation has occurred. Alternatively, in some cases,identifying the deviation may include determining that the abnormalfluid leakage is not present in a particular anatomical region. Forexample, if during a time (or a time interval) of the surgicalprocedure, the abnormal fluid leakage is not present in a particularanatomical region, the method may be configured to determine that thedeviation has occurred.

Aspects of disclosed embodiments may include determining at least oneaction likely to improve a predicted outcome based on accessed frames(e.g., frames of surgical footage), and providing a recommendation basedon the determined at least one action. In various embodiments,determining at least one action may include using a suitable machinelearning method for accessing and analyzing frames of surgicalprocedure. In some examples, a machine learning model may be trainedusing training examples to determine actions likely to improve outcomesof surgical procedures and/or the likely improvements to the outcomesbased on information related to the current state of the surgicalprocedures. An example of such training example may include informationrelated to a state of a particular surgical procedure, together with alabel indicating an action likely to improve an outcome of theparticular surgical procedure, and/or the likely improvement to thepredicted outcome. Such label may be based on an analysis of historicaldata related to historical surgical procedures, on user input, and soforth. Some non-limiting examples of information related to a currentstate of a surgical procedure may include images and/or videos of thesurgical procedure, information based on an analysis of images and/orvideos of the surgical procedure, characteristics of the patientundergoing the surgical procedure, characteristics of a healthcareprofessional performing at least part of the surgical procedure,characteristics of medical instruments used in the surgical procedure,characteristics of an operating room related to the surgical procedure,intraoperative events occurred in the surgical procedure, current time,durations of surgical phases in the surgical procedure, and so forth.Further, in some examples, the trained machine learning model may beused to analyze information related to a current state of the surgicalprocedure, and determining the at least one action likely to improve thepredicted outcome and/or the likely improvement to the predictedoutcome.

Aspects of disclosed embodiments may include providing a recommendationbefore the particular action is performed. The recommendation may be anysuitable electronic notification as described herein and consistent withdisclosed embodiments. Alternatively, the recommendation may be anysuitable sound signal, visual signal, or any other signal (e.g., tactilesignal, such as vibration) that may be transmitted to a healthcareprofessional (e.g., a surgeon administering a surgical procedure).

Various disclosed embodiments may include forgoing providing arecommendation when a likely improvement to a predicted outcome due to adetermined at least one action is below a selected threshold. Forexample, if a likelihood of improvement is below fifty percent, therecommendation may not be provided. In some cases, an improvement of afirst predicted outcome may be offset by an adverse second predictedoutcome, and a recommendation for improving a first predicted outcomemay not be provided. For example, if a first predicted outcome isidentified as “eliminating a rash for a patient” and a second predictedoutcome is identified as a “cardiac arrest,” then even for asufficiently high likelihood of improvement of a first predicted outcome(e.g., ninety-nine percent chance of eliminating a rash for a patient),the recommendation may not be provided due to a second outcome of“cardiac arrest” being possible (even if possibility of the secondoutcome is small, e.g., one percent). Thus, selecting to provide arecommendation or forgoing to provide the recommendation may be based onone or more predicted outcomes. Further, a selected threshold may bebased on one or more selected outcomes. For example, if a first outcomeis an ability for a person to have a possibility of life for the nexttwenty years, and a second adverse outcome is a cardiac arrest, therecommendation may still be provided when the possibility of cardiacarrest is sufficiently low (e.g., lower than thirty percent). In somecases, a threshold may be selected based on a characteristic of thepatient. For example, if a patient is overweight, a selected thresholdfor forgoing providing a recommendation for bariatric surgery may belowered as compared to the same threshold for a less overweight person.In some cases, determining at least one action for improving a predictedoutcome may be further based on a characteristic of a patient. Forexample, if a patient is an elderly person, a bypass procedure may notbe recommended, while such a procedure may be recommended for a youngerperson.

Aspects of embodiments for predicting post discharge risk areillustrated in FIG. 36 by a process 3601. At step 3611, process 3601 mayinclude accessing frames of video captured during a specific surgicalprocedure on a patient using any suitable means. For example, accessingmay occur via a wired or wireless network, via a machine-learning model,or via any other means for allowing reading/writing data. In some cases,accessing frames may include accessing by a healthcare professional. Insuch cases, the healthcare professional may use input devices (e.g.,keyboard, mouse, or any other input device) for accessing the frames.

At step 3613, process 3601 may include accessing stored historical dataidentifying intraoperative events and associated outcomes, as describedabove. At step 3615, process 3601 may include analyzing accessed frames(e.g., frames of surgical footage), and based on information obtainedfrom the historical data, identify in the accessed frames at least onespecific intraoperative event. A process of analyzing the accessedframes and identifying in the accessed frames a specific intraoperativeevent may be performed by a suitable machine-learning model as describedabove.

At step 3617, process 3601 may include determining, based on informationobtained from the historical data and the identified at least oneintraoperative event, a predicted outcome associated with the specificsurgical procedure, as described above. Process 3601 may conclude withstep 3619 for outputting the predicted outcome in a manner associatingthe predicted outcome with the patient, as previously described.

It should be noted that process 3601 is not limited to steps 3611-3619,and new steps may be added, or some of steps 3611-3619 may be replacedor omitted. For example, step 3613 may be omitted.

As previously discussed, the present disclosure relates to a method anda system for predicting post discharge risk, as well as a non-transitorycomputer-readable medium that may include instructions that, whenexecuted by at least one processor, cause the at least one processor toexecute operations enabling predicting post-discharge risk.

Disclosed embodiments may include any one of the followingbullet-pointed features alone or in combination with one or more otherbullet-pointed features, whether implemented as a method, by at leastone processor, and/or stored as executable instructions onnon-transitory computer-readable media:

-   -   accessing at least one video of a surgical procedure    -   causing the at least one video to be output for display    -   overlaying on the at least one video outputted for display a        surgical timeline, wherein the surgical timeline includes        markers identifying at least one of a surgical phase, an        intraoperative surgical event, and a decision making junction    -   enabling a surgeon, while viewing playback of the at least one        video to select one or more markers on the surgical timeline,        and thereby cause a display of the video to skip to a location        associated with the selected marker    -   wherein the markers are coded by at least one of a color or a        criticality level    -   wherein the surgical timeline includes textual information        identifying portions of the surgical procedure    -   wherein the at least one video includes a compilation of footage        from a plurality of surgical procedures, arranged in procedural        chronological order    -   wherein the compilation of footage depicts complications from        the plurality of surgical procedures    -   wherein the one or more markers are associated with the        plurality of surgical procedures and are displayed on a common        timeline    -   wherein the one or more markers include a decision making        junction marker corresponding to a decision making junction of        the surgical procedure    -   wherein the selection of the decision making junction marker        enables the surgeon to view two or more alternative video clips        from two or more corresponding other surgical procedures    -   wherein the two or more video clips present differing conduct    -   wherein the one or more markers include a decision making        junction marker corresponding to a decision making junction of        the surgical procedure    -   wherein the selection of the decision making junction marker        causes a display of one or more alternative possible decisions        related to the selected decision making junction marker    -   wherein one or more estimated outcomes associated with the one        or more alternative possible decisions are displayed in        conjunction with the display of the one or more alternative        possible decisions    -   wherein the one or more estimated outcomes are a result of an        analysis of a plurality of videos of past surgical procedures        including respective similar decision making junctions    -   wherein information related to a distribution of past decisions        made in respective similar past decision making junctions are        displayed in conjunction with the display of the alternative        possible decisions    -   wherein the decision making junction of the surgical procedure        is associated with a first patient, and the respective similar        past decision making junctions are selected from past surgical        procedures associated with patients with similar characteristics        to the first patient    -   wherein the decision making junction of the surgical procedure        is associated with a first medical professional, and the        respective similar past decision making junctions are selected        from past surgical procedures associated with medical        professionals with similar characteristics to the first medical        professional    -   wherein the decision making junction of the surgical procedure        is associated with a first prior event in the surgical        procedure, and the similar past decision making junctions are        selected from past surgical procedures including prior events        similar to the first prior event    -   wherein the markers include intraoperative surgical event        markers    -   wherein selection of an intraoperative surgical event marker        enables the surgeon to view alternative video dips from        differing surgical procedures    -   wherein the alternative video clips present differing ways in        which a selected intraoperative surgical event was handled    -   wherein the overlay on the video output is displayed before the        end of the surgical procedure depicted in the displayed video    -   wherein the analysis is based on one or more electronic medical        records associated with the plurality of videos of past surgical        procedures    -   wherein the respective similar decision making junctions are        similar to the decision making junction of the surgical        procedure according to a similarity metric    -   wherein the analysis includes usage of an implementation of a        computer vision algorithm    -   wherein the markers relate to intraoperative surgical events and        the selection of an intraoperative surgical event marker enables        the surgeon to view alternative video clips from differing        surgical procedures    -   accessing video footage to be indexed, the video footage to be        indexed including footage of a particular surgical procedure    -   analyzing the video footage to identify a video footage location        associated with a surgical phase of the particular surgical        procedure    -   generating a phase tag associated with the surgical phase    -   associating the phase tag with the video footage location    -   analyzing the video footage to identify an event location of a        particular intraoperative surgical event within the surgical        phase    -   associating an event tag with the event location of the        particular intraoperative surgical event    -   storing an event characteristic associated with the particular        intraoperative surgical event    -   associating at least a portion of the video footage of the        particular surgical procedure with the phase tag, the event tag,        and the event characteristic in a data structure that contains        additional video footage of other surgical procedures    -   wherein the data structure also includes respective phase tags,        respective event tags, and respective event characteristics        associated with one or more of the other surgical procedures    -   enabling a user to access the data structure through selection        of a selected phase tag, a selected event tag, and a selected        event characteristic of video footage for display    -   performing a lookup in the data structure of surgical video        footage matching the at least one selected phase tag, selected        event tag, and selected event characteristic to identify a        matching subset of stored video footage    -   causing the matching subset of stored video footage to be        displayed to the user, to thereby enable the user to view        surgical footage of at least one intraoperative surgical event        sharing the selected event characteristic, while omitting        playback of video footage lacking the selected event        characteristic    -   wherein enabling the user to view surgical footage of at least        one intraoperative surgical event that has the selected event        characteristic, while omitting playback of portions of selected        surgical events lacking the selected event characteristic,        includes sequentially presenting to the user portions of        surgical footage of a plurality of intraoperative surgical        events sharing the selected event characteristic, while omitting        playback of portions of selected surgical events lacking the        selected event characteristic    -   wherein the stored event characteristic includes an adverse        outcome of the surgical event    -   wherein causing the matching subset to be displayed includes        enabling the user to view surgical footage of a selected adverse        outcome while omitting playback of surgical events lacking the        selected adverse outcome    -   wherein the stored event characteristic includes a surgical        technique    -   wherein causing the matching subset to be displayed includes        enabling the user to view surgical footage of a selected        surgical technique while omitting playback of surgical footage        not associated with the selected surgical technique    -   wherein the stored event characteristic includes a surgeon skill        level    -   wherein causing the matching subset to be displayed includes        enabling the user to view footage exhibiting a selected surgeon        skill level while omitting playback of footage lacking the        selected surgeon skill level    -   wherein the stored event characteristic includes a physical        patient characteristic    -   wherein causing the matching subset to be displayed includes        enabling the user to view footage exhibiting a selected physical        patient characteristic while omitting playback of footage        lacking the selected physical patient characteristic    -   wherein the stored event characteristic includes an identity of        a specific surgeon    -   wherein causing the matching subset to be displayed includes        enabling the user to view footage exhibiting an activity by a        selected surgeon while omitting playback of footage lacking        activity by the selected surgeon    -   wherein the stored event characteristic includes physiological        response    -   wherein causing the matching subset to be displayed includes        enabling the user to view footage exhibiting a selected        physiological response while omitting playback of footage        lacking the selected physiological response    -   wherein analyzing the video footage to identify the video        footage location associated with at least one of the surgical        event or the surgical phase includes performing computer image        analysis on the video footage to identify at least one of a        beginning location of the surgical phase for playback or a        beginning of a surgical event for playback    -   accessing aggregate data related to a plurality of surgical        procedures similar to the particular surgical procedure    -   presenting to the user statistical information associated with        the selected event characteristic    -   wherein the accessed video footage includes video footage        captured via at least one image sensor located in at least one        of a position above an operating table, in a surgical cavity of        a patient, within an organ of a patient or within vasculature of        a patient    -   wherein identifying the video footage location is based on user        input    -   wherein identifying the video footage location includes using        computer analysis to analyze frames of the video footage    -   wherein the computer image analysis includes using a neural        network model trained using example video frames including        previously-identified surgical phases to thereby identify at        least one of a video footage location or a phase tag    -   determining the stored event characteristic based on user input    -   determining the stored event characteristic based on a computer        analysis of video footage depicting the particular        intraoperative surgical event    -   wherein generating the phase tag is based on a computer analysis        of video footage depicting the surgical phase    -   wherein identifying a matching subset of stored video footage        includes using computer analysis to determine a degree of        similarity between the matching subset of stored video and the        selected event characteristic    -   accessing particular surgical footage containing a first group        of frames associated with at least one intraoperative surgical        event and a second group of frames not associated with surgical        activity    -   accessing historical data based on historical surgical footage        of prior surgical procedures, wherein the historical data        includes information that distinguishes portions of surgical        footage into frames associated with intraoperative surgical        events and frames not associated with surgical activity    -   distinguishing in the particular surgical footage the first        group of frames from the second group of frames based on the        information of the historical data    -   upon request of a user, presenting to the user an aggregate of        the first group of frames of the particular surgical footage,        while omitting presentation to the user of the second group of        frames    -   wherein the information that distinguishes portions of the        historical surgical footage into frames associated with an        intraoperative surgical event includes an indicator of at least        one of a presence or a movement of a surgical tool    -   wherein the information that distinguishes portions of the        historical surgical footage into frames associated with an        intraoperative surgical event includes detected tools and        anatomical features in associated frames    -   wherein the request of the user includes an indication of at        least one type of intraoperative surgical event of interest, and    -   wherein the first group of frames depicts at least one        intraoperative surgical event of the at least one type of        intraoperative surgical event of interest    -   wherein the request of the user includes a request to view a        plurality of intraoperative surgical events in the particular        surgical footage, and    -   wherein presenting to the user an aggregate of the first group        of frames includes displaying the first group frames in        chronological order with chronological frames of the second        group omitted    -   wherein the historical data further includes historical surgical        outcome data and respective historical cause data    -   wherein the first group of frames includes a cause set of frames        and an outcome set of frames    -   wherein the second group of frames includes an intermediate set        of frames    -   analyzing the particular surgical footage to identify a surgical        outcome and a respective cause of the surgical outcome, the        identifying being based on the historical outcome data and        respective historical cause data    -   detecting, based on the analyzing, the outcome set of frames in        the particular surgical footage, the outcome set of frames being        within an outcome phase of the surgical procedure    -   detecting, based on the analyzing, a cause set of frames in the        particular surgical footage, the cause set of frames being        within a cause phase of the surgical procedure remote in time        from the outcome phase    -   wherein the intermediate set of frames is within an intermediate        phase interposed between the cause set of frames and the outcome        set of frames    -   generating a cause-effect summary of the surgical footage    -   wherein the cause-effect summary includes the cause set of        frames and the outcome set of frames and omits the intermediate        set of frames    -   wherein the aggregate of the first group of frames presented to        the user includes the cause-effect summary    -   wherein the cause phase includes a surgical phase in which the        cause occurred    -   wherein the cause set of frames is a subset of the frames in the        cause phase    -   wherein the outcome phase includes a surgical phase in which the        outcome is observable    -   wherein the outcome set of frames is a subset of frames in the        outcome phase    -   using a machine learning model trained to identify surgical        outcomes and respective causes of the surgical outcomes using        the historical data to analyze the particular surgical footage    -   wherein the particular surgical footage depicts a surgical        procedure performed on a patient and captured by at least one        image sensor in an operating room    -   exporting the first group of frames for storage in a medical        record of the patient    -   generating an index of the at least one intraoperative surgical        event, and exporting the first group of frames includes        generating a compilation of the first group of frames, the        compilation including the index and being configured to enable        viewing of the at least one intraoperative surgical event based        on a selection of one or more index items    -   wherein the compilation contains a series of frames of differing        intraoperative events stored as a continuous video    -   associating the first group of frames with a unique patient        identifier and updating a medical record including the unique        patient identifier    -   wherein a location of the at least one image sensor is at least        one of above an operating table in the operating room or within        the patient    -   wherein distinguishing in the particular surgical footage the        first group of frames from the second group of frames includes:        analyzing the particular surgical footage to detect a medical        instrument    -   analyzing the particular surgical footage to detect an        anatomical structure    -   analyzing the video to detect a relative movement between the        detected medical instrument and the detected anatomical        structure    -   distinguishing the first group of frames from the second group        of frames based on the relative movement    -   wherein the first group of frames includes surgical activity        frames and the second group of frames includes non-surgical        activity frames    -   wherein presenting the aggregate thereby enables a surgeon        preparing for surgery to omit the non-surgical activity frames        during a video review of the abridged presentation    -   wherein distinguishing the first group of frames from the second        group of frames is further based on a detected relative position        between the medical instrument and the anatomical structure    -   wherein distinguishing the first group of frames from the second        group of frames is further based on a detected interaction        between the medical instrument and the anatomical structure    -   wherein omitting the non-surgical activity frames includes        omitting a majority of frames that capture non-surgical activity    -   accessing a repository of a plurality of sets of surgical video        footage reflecting a plurality of surgical procedures performed        on differing patients and including intraoperative surgical        events, surgical outcomes, patient characteristics, surgeon        characteristics, and intraoperative surgical event        characteristics    -   enabling a surgeon preparing for a contemplated surgical        procedure to input case-specific information corresponding to        the contemplated surgical procedure    -   comparing the case-specific information with data associated        with the plurality of sets of surgical video footage to identify        a group of intraoperative events likely to be encountered during        the contemplated surgical procedure    -   using the case-specific information and the identified group of        intraoperative events likely to be encountered to identify        specific frames in specific sets of the plurality of sets of        surgical video footage corresponding to the identified group of        intraoperative events    -   wherein the identified specific frames include frames from the        plurality of surgical procedures performed on differing patients    -   determining that a first set and a second set of video footage        from differing patients contain frames associated with        intraoperative events sharing a common characteristic    -   omitting an inclusion of the second set from a compilation to be        presented to the surgeon and including the first set in the        compilation to be presented to the surgeon    -   enabling the surgeon to view a presentation including the        compilation containing frames from the differing surgical        procedures performed on differing patients    -   enabling a display of a common surgical timeline including one        or more chronological markers corresponding to one or more of        the identified specific frames along the presentation    -   wherein enabling the surgeon to view the presentation includes        sequentially displaying discrete sets of video footage of the        differing surgical procedures performed on differing patients    -   wherein sequentially displaying discrete sets of video footage        includes displaying an index of the discrete sets of video        footage enabling the surgeon to select one or more of the        discrete sets of video footage    -   wherein the index includes a timeline parsing the discrete sets        into corresponding surgical phases and textual phase indicators    -   wherein the timeline includes an intraoperative surgical event        marker corresponding to an intraoperative surgical event    -   wherein the surgeon is enabled to click on the intraoperative        surgical event marker to display at least one frame depicting        the corresponding intraoperative surgical event    -   wherein the case-specific information corresponding to the        contemplated surgical procedure is received from an external        device    -   wherein comparing the case-specific information with data        associated with the plurality of sets of surgical video footage        includes using an artificial neural network to identify the        group of intraoperative events likely to be encountered during        the contemplated surgical procedure    -   wherein using the artificial neural network includes providing        the case-specific information to the artificial neural network        as an input    -   wherein the case-specific information includes a characteristic        of a patient associated with the contemplated procedure    -   wherein the characteristic of the patient is received from a        medical record of the patient    -   wherein the case-specific information includes information        relating to a surgical tool    -   where the information relating to the surgical tool includes at        least one of a tool type or a tool model    -   wherein the common characteristic includes a characteristic of        the differing patients    -   wherein the common characteristic includes an intraoperative        surgical event characteristic of the contemplated surgical        procedure    -   wherein determining that a first set and a second set of video        footage from differing patients contain frames associated with        intraoperative events sharing a common characteristic includes        using an implementation of a machine learning model to identify        the common characteristic    -   using example video footage to train the machine learning model        to determine whether two sets of video footage share the common        characteristic    -   wherein implementing the machine learning model includes        implementing the trained machine learning model    -   training a machine learning model to generate an index of the        repository based on the intraoperative surgical events, the        surgical outcomes, the patient characteristics, the surgeon        characteristics, and the intraoperative surgical event        characteristic    -   generating the index of the repository    -   wherein comparing the case-specific information with data        associated with the plurality of sets includes searching the        index    -   analyzing frames of the surgical footage to identify in a first        set of frames an anatomical structure    -   accessing first historical data, the first historical data being        based on an analysis of first frame data captured from a first        group of prior surgical procedures    -   analyzing the first set of frames using the first historical        data and using the identified anatomical structure to determine        a first surgical complexity level associated with the first set        of frames    -   analyzing frames of the surgical footage to identify in a second        set of frames a medical tool, the anatomical structure, and an        interaction between the medical tool and the anatomical        structure    -   accessing second historical data, the second historical data        being based on an analysis of a second frame data captured from        a second group of prior surgical procedures    -   analyzing the second set of frames using the second historical        data and using the identified interaction to determine a second        surgical complexity level associated with the second set of        frames    -   wherein determining the first surgical complexity level further        includes identifying in the first set of frames a medical tool    -   wherein determining the second surgical complexity level is        based on time elapsed from the first set of frames to the second        set of frames    -   wherein at least one of determining the first complexity level        or second complexity level is based on a physiological response    -   determining a level of skill demonstrated by a healthcare        provider in the surgical footage    -   wherein at least one of determining the first complexity level        or second complexity level is based on the determined level of        skill demonstrated by the healthcare provider    -   determining that the first surgical complexity level is less        than a selected threshold, determining that the second surgical        complexity level exceeds the selected threshold, and in response        to the determination that the first surgical complexity level is        less than the selected threshold and the determination that the        second surgical complexity level exceeds the selected threshold,        storing the second set of frames in a data structure while        omitting the first set of frames from the data structure    -   wherein identifying the anatomical structure in the first set of        frames is based on an identification of a medical tool and a        first interaction between the medical tool and the anatomical        structure    -   tagging the first set of frames with the first surgical        complexity level    -   tagging the second set of frames with the second surgical        complexity level    -   generating a data structure including the first set of frames        with the first tag and the second set of frames with the second        tag to enable a surgeon to select the second surgical complexity        level, and thereby cause the second set of frames to be        displayed, while omitting a display of the first set of frames    -   using a machine learning model trained to identify surgical        complexity levels using frame data captured from prior surgical        procedures to determine at least one of the first surgical        complexity level or the second surgical complexity level    -   wherein determining the second surgical complexity level is        based on an event that occurred between the first set of frames        and the second set of frames    -   wherein determining at least one of the first surgical        complexity level or the second surgical complexity level is        based on a condition of the anatomical structure    -   wherein determining at least one of the first surgical        complexity level or the second surgical complexity level is        based on an analysis of an electronic medical record    -   wherein determining the first surgical complexity level is based        on an event that occurred after the first set of frames    -   wherein determining at least one of the first surgical        complexity level or the second surgical complexity level is        based on a skill level of a surgeon associated with the surgical        footage    -   wherein determining the second surgical complexity level is        based on an indication that an additional surgeon was called        after the first set of frames    -   wherein determining the second surgical complexity level is        based on an indication that a particular medicine was        administered after the first set of frames    -   wherein the first historical data includes a machine learning        model trained using the first frame data captured from the first        group of prior surgical procedures    -   wherein the first historical data includes an indication of a        statistical relation between a particular anatomical structure        and a particular surgical complexity level    -   receiving, from an image sensor positioned in a surgical        operating room, visual data tracking an ongoing surgical        procedure    -   accessing a data structure containing information based on        historical surgical data    -   analyzing the visual data of the ongoing surgical procedure        using the data structure to determine an estimated completion        time of the ongoing surgical procedure    -   accessing a schedule for the surgical operating room including a        scheduled time associated with completion of the ongoing        surgical procedure    -   calculating, based on the estimated completion time of the        ongoing surgical procedure, whether an expected time of        completion is likely to result in variance from the scheduled        time associated with the completion    -   outputting a notification upon calculation of the variance, to        thereby enable subsequent users of the surgical operating room        to adjust their schedules accordingly    -   wherein the notification includes an updated operating room        schedule    -   wherein the updated operating room schedule enables a queued        healthcare professional to prepare for a subsequent surgical        procedure    -   electronically transmitting the notification to a device        associated with a subsequent scheduled user of the surgical        operating room    -   determining an extent of the variance from the scheduled time        associated with the completion    -   in response to a first determined extent, outputting the        notification    -   in response to a second determined extent, forgoing outputting        the notification    -   determining whether the expected time of completion is likely to        result in a delay of at least a selected threshold amount of        time from the scheduled time associated with the completion    -   in response to a determination that the expected time of        completion is likely to result in a delay of at least the        selected threshold amount of time, outputting the notification    -   in response to a determination that the expected time of        completion is not likely to result in a delay of at least the        selected threshold amount of time, forgoing outputting the        notification    -   wherein the determining the estimated completion time is based        on one or more stored characteristics associated with a        healthcare professional conducting the ongoing surgical        procedure    -   updating a historical average time to completion based on        determined actual time to complete the ongoing surgical        procedure    -   wherein the image sensor is positioned above a patient    -   wherein the image sensor is positioned on a surgical tool    -   wherein analyzing further includes detecting a characteristic        event in the received visual data, assessing the information        based on historical surgical data to determine an expected time        to complete the surgical procedure following an occurrence of        the characteristic event in the historical surgical data, and        determining the estimated completion time based on the        determined expected time to complete    -   using historical visual data to train a machine learning model        to detect the characteristic event    -   using historical visual data to train a machine learning model        to estimate completion times    -   wherein calculating the estimated completion time includes        implementing the trained machine learning model trained    -   using average historical completion times to determine the        estimated completion time    -   detecting a medical tool in the visual data    -   wherein calculating the estimated completion time is based on        the detected medical tool    -   wherein analyzing further includes detecting an anatomical        structure in the visual data    -   wherein calculating the estimated completion time is based on        the detected anatomical structure    -   wherein analyzing further includes detecting an interaction        between an anatomical structure and a medical tool in the visual        data    -   wherein calculating the estimated completion time is based on        the detected interaction    -   wherein analyzing further includes determining a skill level of        a surgeon in the visual data    -   wherein calculating the estimated completion time is based on        the determined skill level    -   accessing video frames captured during a surgical procedure on a        patient    -   analyzing the video frames captured during the surgical        procedure to identify in the video frames at least one medical        instrument, at least one anatomical structure, and at least one        interaction between the at least one medical instrument and the        at least one anatomical structure    -   accessing a database of reimbursement codes correlated to        medical instruments, anatomical structures, and interactions        between medical instruments and anatomical structures    -   comparing the identified at least one interaction between the at        least one medical instrument and the at least one anatomical        structure with information in the database of reimbursement        codes to determine at least one reimbursement code associated        with the surgical procedure    -   outputting the at least one reimbursement code for use in        obtaining an insurance reimbursement for the surgical procedure    -   wherein the at least one reimbursement code outputted includes a        plurality of outputted reimbursement codes    -   wherein at least two of the plurality of outputted reimbursement        codes are based on differing interactions with a common        anatomical structure    -   wherein the at least two outputted reimbursement codes are        determined based in part on detection of two differing medical        instruments    -   wherein determining the at least one reimbursement code is also        based on an analysis of a postoperative surgical report    -   wherein the video frames are captured from an image sensor        positioned above the patient    -   wherein the video frames are captured from an image sensor        associated with a medical device    -   updating the database by associating the at least one        reimbursement code with the surgical procedure    -   generating correlations between processed reimbursement codes        and at least one of a plurality of medical instruments in        historical video footage, a plurality of anatomical structures        in the historical video footage, or a plurality of interactions        between medical instruments and anatomical structures in the        historical video footage    -   updating the database based on the generated correlations    -   wherein generating correlations includes implementing a        statistical model    -   using a machine learning model to detect, in the historical        video footage, the at least one plurality of medical        instruments, plurality of anatomical structures, or plurality of        interactions between medical instruments and anatomical        structures    -   analyzing the video frames captured during the surgical        procedure to determine a condition of an anatomical structure of        the patient    -   determining the at least one reimbursement code associated with        the surgical procedure based on the determined condition of the        anatomical structure    -   analyzing the video frames captured during the surgical        procedure to determine a change in a condition of an anatomical        structure of the patient during the surgical procedure    -   determining the at least one reimbursement code associated with        the surgical procedure based on the determined change in the        condition of the anatomical structure    -   analyzing the video frames captured during the surgical        procedure to determine a usage of a particular medical device    -   determining the at least one reimbursement code associated with        the surgical procedure based on the determined usage of the        particular medical device    -   analyzing the video frames captured during the surgical        procedure to determine a type of usage of the particular medical        device    -   in response to a first determined type of usage, determining at        least a first reimbursement code associated with the surgical        procedure    -   in response to a second determined type of usage, determining at        least a second reimbursement code associated with the surgical        procedure, the at least a first reimbursement code differing        from the at least a second reimbursement code    -   receiving a processed reimbursement code associated with the        surgical procedure, and updating the database based on the        processed reimbursement code    -   wherein the processed reimbursement code differs from a        corresponding reimbursement code of the at least one        reimbursement codes    -   analyzing the video frames captured during the surgical        procedure to determine an amount of a medical supply of a        particular type used in the surgical procedure    -   determining the at least one reimbursement code associated with        the surgical procedure based on the determined amount    -   receiving an input of an identifier of a patient    -   receiving an input of an identifier of a health care provider    -   receiving an input of surgical footage of a surgical procedure        performed on the patient by the health care provider    -   analyzing a plurality of frames of the surgical footage to        derive image-based information for populating a post-operative        report of the surgical procedure    -   causing the derived image-based information to populate the        post-operative report of the surgical procedure    -   analyzing the surgical footage to identify one or more phases of        the surgical procedure and to identify a property of at least        one phase of the identified phases    -   wherein the derived image-based information is based on the        identified at least one phase and the identified property of the        at least one phase    -   analyzing the surgical footage to associate a name with the at        least one phase    -   wherein the derived image-based information includes the name        associated with the at least one phase    -   determining at least a beginning of the at least one phase    -   wherein the derived image-based information is based on the        determined beginning    -   associating a time marker with the at least one phase    -   wherein the derived image-based information includes the time        marker associated with the at least one phase    -   transmitting data to the health care provider, the transmitted        data including the patient identifier and the derived        image-based information    -   analyzing the surgical footage to identify at least one        recommendation for post-operative treatment    -   providing the identified at least one recommendation    -   wherein the caused populating of the post-operative report of        the surgical procedure is configured to enable the health care        provider to alter at least part of the derived image-based        information in the post-operative report    -   wherein the caused populating of the post-operative report of        the surgical procedure is configured to cause at least part of        the derived image-based information to be identified in the        post-operative report as automatically generated data    -   analyzing the surgical footage to identify a surgical event        within the surgical footage and to identify a property of the        identified surgical event    -   wherein the derived image-based information is based on the at        identified surgical event and the identified property    -   analyzing the surgical footage to determine an event name of the        identified surgical event    -   wherein the derived image-based information includes the        determined event name    -   associating a time marker with the identified surgical event    -   wherein the derived image-based information includes the time        marker    -   providing the derived image-based information in a form enabling        updating of an electronic medical record    -   wherein the derived image-based information is based in part on        user input    -   wherein the derived image-based information includes a first        part associated with a first portion of the surgical procedure        and a second part associated with a second portion of the        surgical procedure, and further including: receiving a        preliminary post-operative report    -   analyzing the preliminary post-operative report to select a        first position and a second position within the preliminary        post-operative report, the first position is associated with the        first portion of the surgical procedure and the second position        is associated with the second portion of the surgical procedure    -   causing the first part of the derived image-based information to        be inserted at the selected first position and the second part        of the derived image-based information to be inserted at the        selected second position    -   analyzing the surgical footage to select at least part of at        least one frame of the surgical footage    -   causing the selected at least part of at least one frame of the        surgical footage to be included in the post-operative report of        the surgical procedure    -   receiving a preliminary post-operative report    -   analyzing the preliminary post-operative report and the surgical        footage to select the at least part of at least one frame of the        surgical footage    -   receiving a preliminary post-operative report    -   analyzing the preliminary post-operative report and the surgical        footage to identify at least one inconsistency between the        preliminary post-operative report and the surgical footage    -   providing an indication of the identified at least one        inconsistency    -   accessing frames of video captured during a specific surgical        procedure    -   accessing stored data identifying a recommended sequence of        events for the surgical procedure    -   comparing the accessed frames with the recommended sequence of        events to identify an indication of a deviation between the        specific surgical procedure and the recommended sequence of        events for the surgical procedure    -   determining a name of an intraoperative surgical event        associated with the deviation    -   providing a notification of the deviation including the name of        the intraoperative surgical event associated with the deviation    -   wherein identifying the indication of the deviation and        providing the notification occurs in real time during the        surgical procedure    -   receiving an indication that a particular action is about to        occur in the specific surgical procedure    -   identifying, using the recommended sequence of events, a        preliminary action to the particular action    -   determining, based on an analysis of the accessed frames, that        the identified preliminary action did not yet occurred    -   in response to the determination that the identified preliminary        action did not yet occur identifying the indication of the        deviation    -   wherein the specific surgical procedure is a cholecystectomy    -   wherein the recommended sequence of events is based on a        critical view of safety    -   wherein the specific surgical procedure is an appendectomy    -   wherein the specific surgical procedure is a hernia repair    -   wherein the specific surgical procedure is a hysterectomy    -   wherein the specific surgical procedure is a radical        prostatectomy    -   wherein the specific surgical procedure is a partial        nephrectomy, and the deviation includes neglecting to identify a        renal hilum    -   wherein the specific surgical procedure is a thyroidectomy, and        the deviation includes neglecting to identify a recurrent        laryngeal nerve    -   identifying a set of frames associated with the deviation    -   wherein providing the notification includes displaying the        identified set of frames associated with the deviation    -   wherein the indication that the particular action is about to        occur is based on an input from a surgeon performing the        specific surgical procedure    -   wherein the indication that the particular action is about to        occur is an entrance of a particular medical instrument to a        selected region of interest    -   wherein identifying the deviation includes determining that a        surgical tool is in a particular anatomical region    -   wherein the specific surgical procedure is a hemicolectomy    -   wherein the deviation includes neglecting to perform an        anastomosis    -   where identifying the indication of the deviation is based on an        elapsed time associated with an intraoperative surgical        procedure    -   receiving video footage of a surgical procedure performed by a        surgeon on a patient in an operating room    -   accessing at least one data structure including image-related        data characterizing surgical procedures    -   analyzing the received video footage using the image-related        data to determine an existence of a surgical decision making        junction    -   accessing, in the at least one data structure, a correlation        between an outcome and a specific action taken at the decision        making junction    -   based on the determined existence of the decision making        junction and the accessed correlation, outputting a        recommendation to a user to undertake the specific action    -   wherein the instructions are configured to cause the at least        one processor to execute the operations in real time during the        surgical procedure    -   wherein the user is the surgeon    -   wherein the decision making junction is determined by an        analysis of a plurality of differing historical procedures where        differing courses of action occurred following a common surgical        situation    -   wherein the video footage includes images from at least one of        an endoscope and an intracorporeal camera    -   wherein the recommendation includes a recommendation to conduct        a medical test    -   receiving a result of the medical test    -   based on the determined existence of the decision making        junction, the accessed correlation and the received result of        the medical test, outputting a second recommendation to the user        to undertake a particular action    -   wherein the specific action includes brining an additional        surgeon to the operating room    -   wherein the decision making junction includes at least one of        inappropriate access or exposure, retraction of an anatomical        structure, misinterpretation of an anatomical structure or a        fluid leak    -   wherein the recommendation includes a confidence level that a        desired surgical outcome will occur if the specific action is        taken    -   wherein the recommendation includes a confidence level that a        desired outcome will not occur if the specific action is not        taken    -   wherein the recommendation is based on time elapsed since a        particular point in the surgical procedure    -   wherein the recommendation includes an indication of an        undesired surgical outcome likely to occur if the specific        action is not undertaken    -   wherein the recommendation is based on a skill level of the        surgeon    -   wherein the recommendation is based on a surgical event that        occurred in the surgical procedure prior to the decision making        junction    -   wherein the specific action includes a plurality of steps    -   wherein the determination of the existence of the surgical        decision making junction is based on at least one of a detected        physiological response of an anatomical structure and a motion        associated with a surgical tool    -   receiving a vital sign of the patient and    -   wherein the recommendation is based on the accessed correlation        and the vital sign    -   wherein the surgeon is a surgical robot and the recommendation        is provided in the form of an instruction to the surgical robot    -   wherein the recommendation is based on a condition of a tissue        of the patient    -   wherein the recommendation of the specific action includes a        creation of a stoma    -   receiving, from at least one image sensor in an operating room,        image data of a surgical procedure    -   analyzing the received image data to determine an identity of an        anatomical structure and to determine a condition of the        anatomical structure as reflected in the image data    -   selecting a contact force threshold associated with the        anatomical structure, the selected contact force threshold being        based on the determined condition of the anatomical structure    -   receiving an indication of actual contact force on the        anatomical structure    -   comparing the indication of actual contact force with the        selected contact force threshold    -   outputting a notification based on a determination that the        indication of actual contact force exceeds the selected contact        force threshold    -   wherein the contact force threshold is associated with a tension        level    -   wherein the contact force threshold is associated with a        compression level    -   wherein the actual contact force is associated with a contact        between a medical instrument and the anatomical structure    -   wherein the indication of actual contact force is estimated        based on an image analysis of the image data    -   wherein outputting the notification includes providing a real        time warning to a surgeon conducting the surgical procedure    -   wherein the notification is an instruction to a surgical robot    -   determining from the image data that the surgical procedure is        in a fight mode    -   wherein the notification is suspended during the fight mode    -   determining from the image data that the surgeon is operating in        a mode ignoring contact force notifications, and suspending at        least temporarily, further contact force notifications based on        the determination that the surgeon is operating in the mode        ignoring contact force notifications    -   wherein selecting the contact force threshold is based on a        location of contact between the anatomical structure and a        medical instrument    -   wherein selecting the contact force threshold is based on an        angle of contact between the anatomical structure and a medical        instrument    -   wherein selecting the contact force threshold includes providing        the condition of the anatomical structure to a regression model        as an input, and selecting the contact force threshold based on        an output of the regression model    -   wherein selecting the contact force threshold is based on a        table of anatomical structures including corresponding contact        force thresholds    -   wherein selecting the contact force threshold is based on        actions performed by a surgeon    -   wherein the indication of actual contact force is received from        a surgical tool    -   wherein the indication of actual contact force is received from        a surgical robot    -   using a machine learning model trained using training examples        to determine the condition of the anatomical structure in the        image data    -   using a machine learning model trained using training examples        to select the contact force threshold    -   receiving, from at least one image sensor arranged to capture        images of a surgical procedure, image data associated with a        first event during the surgical procedure    -   determining, based on the received image data associated with        the first event, a predicted outcome associated with the        surgical procedure    -   receiving, from at least one image sensor arranged to capture        images of a surgical procedure, image data associated with a        second event during the surgical procedure    -   determining, based on the received image data associated with        the second event, a change in the predicted outcome, causing the        predicted outcome to drop below a threshold    -   accessing a data structure of image-related data based on prior        surgical procedures    -   identifying. based on the accessed image-related data, a        recommended remedial action    -   outputting the recommended remedial action    -   wherein the recommended remedial action includes a        recommendation for a surgeon to take a break from the surgical        procedure    -   wherein the recommended remedial action includes a        recommendation to request assistance from another surgeon    -   wherein the recommended remedial action includes a revision to        the surgical procedure    -   wherein the predicted outcome includes a likelihood of hospital        readmission    -   wherein determining the change in the predicted outcome is based        on a magnitude of bleeding    -   wherein identifying the remedial action is based on an        indication that the remedial action is likely to raise the        predicted outcome above the threshold    -   wherein identifying the remedial action includes using a machine        learning model trained to identify remedial actions using        historical examples of remedial actions and surgical outcomes    -   wherein determining the predicted outcome includes using a        machine learning model trained to determine predicted outcomes        based on historical surgical videos and information indicating        surgical outcome corresponding to the historical surgical videos    -   wherein determining the predicted outcome includes identifying        an interaction between a surgical tool and an anatomical        structure, and determining the predicted outcome based on the        identified interaction    -   wherein determining the predicted outcome is based on a skill        level of a surgeon depicted in the image data    -   determining a skill level of a surgeon depicted in the image        data    -   wherein determining the change in the predicted outcome is based        on the skill level    -   further including, in response to the predicted outcome dropping        below a threshold, updating a scheduling record associated with        a surgical room related to the surgical procedure    -   wherein determining the change in the predicted outcome is based        on a time elapsed between a particular point in the surgical        procedure and the second event    -   wherein determining the predicted outcome is based on a        condition of an anatomical structure depicted in the image data    -   determining the condition of the anatomical structure    -   wherein determining the change in the predicted outcome is based        on a change of a color of at least part of the anatomical        structure    -   wherein determining the change in the predicted outcome is based        on a change of appearance of at least part of the anatomical        structure    -   receiving in real time, intracavitary video of a surgical        procedure    -   analyzing frames of the intracavitary video to determine an        abnormal fluid leakage situation in the intracavitary video    -   instituting a remedial action when the abnormal fluid leakage        situation is determined    -   wherein the fluid includes at least one of blood, bile or urine    -   wherein analyzing includes analyzing the frames of the        intracavitary video to identify a blood splash and at least one        property of the blood splash    -   wherein a selection of the remedial action depends on the at        least one property of the identified blood splash    -   wherein the at least one property is associated with a source of        the blood splash    -   wherein the at least one property is associated with an        intensity of the blood splash    -   wherein the at least one property is associated with a volume of        the blood splash    -   wherein analyzing the frames of the intracavitary video includes        determining a property of the abnormal fluid leakage situation    -   wherein a selection of the remedial action depends on the        determined property    -   wherein the property is associated with a volume of the fluid        leakage    -   wherein the property is associated with a color of the fluid        leakage    -   wherein the property is associated with a type of fluid        associated with the fluid leakage    -   wherein the property is associated with a fluid leakage rate    -   storing the intracavitary video, and, upon determining the        abnormal leakage situation, analyzing prior frames of the stored        intracavitary video to determine a leakage source    -   wherein instituting the remedial action includes providing a        notification of a leakage source    -   wherein determining the leakage source includes identifying a        ruptured anatomical organ    -   determining a flow rate associated with the fluid leakage        situation    -   wherein instituting the remedial action is based on the flow        rate    -   determining a volume of fluid loss associated with the fluid        leakage situation    -   wherein instituting the remedial action is based on the volume        of fluid loss    -   wherein analyzing frames of intracavitary video to determine an        abnormal fluid leakage situation in intracavitary video includes        determining whether the determined fluid leakage situation is an        abnormal fluid leakage situation, and    -   in response to a determination that the determined fluid leakage        situation is an abnormal fluid leakage situation, instituting        the remedial action    -   in response to a determination that the determined fluid leakage        situation is normal fluid leakage situation, forgoing        institution of the remedial action    -   wherein the intracavitary video depicts a surgical robot        performing the surgical procedure, and the remedial action        includes sending instructions to the robot    -   accessing frames of video captured during a specific surgical        procedure on a patient    -   accessing stored historical data identifying intraoperative        events and associated outcomes    -   analyzing the accessed frames, and based on information obtained        from the historical data, identifying in the accessed frames at        least one specific intraoperative event    -   determining, based on information obtained from the historical        data and the identified at least one intraoperative event, a        predicted outcome associated with the specific surgical        procedure    -   outputting the predicted outcome in a manner associating the        predicted outcome with the patient    -   wherein identifying the at least one specific intraoperative        event is based on at least one of a detected surgical tool in        the accessed frames, a detected anatomical structure in the        accessed frames, an interaction in the accessed frames between a        surgical tool and an anatomical structure, or a detected        abnormal fluid leakage situation in the accessed frames    -   wherein a machine learning model is used to identify in the        accessed frames the at least one specific intraoperative event,        the machine learning model trained using example training data    -   wherein determining the predicted outcome is based on at least        one of a characteristic of the patient, an electronic medical        record, or a postoperative surgical report    -   wherein a machine learning model is used to determine the        predicted outcome associated with the specific surgical        procedure based on intraoperative events, the machine learning        model trained using training examples    -   wherein determining a predicted outcome includes using the        trained machine learning model to predict surgical outcomes        based on the identified intraoperative event and an identified        characteristic of the patient    -   receiving information identifying a realized surgical outcome        following the surgical procedure and updating the machine        learning model by training the machine learning model using the        received information    -   identifying a characteristic of the patient    -   wherein the predicted outcome is also determined based on the        identified patient characteristic    -   wherein the patient characteristic is derived from an electronic        medical record    -   wherein identifying the patient characteristic includes using a        machine learning model to analyze the accessed frames, the        machine learning model being trained to identify patient        characteristics using training examples of historical surgical        procedures and corresponding historical patient characteristics    -   wherein the predicted outcome includes at least one of a        post-discharge mishap, a post-discharge adverse event, a        post-discharge complication, or an estimate of a risk of        readmission    -   accessing a data structure containing recommended sequences of        surgical events    -   wherein identifying the at least one specific intraoperative        event is based on an identification of a deviation between a        recommended sequence of events for the surgical procedure        identified in the data structure, and an actual sequence of        events detected in the accessed frames    -   wherein the identification of the deviation is based on at least        one of a detected surgical tool in the accessed frames, a        detected anatomical structure in the accessed frames, or an        interaction in the accessed frames between a surgical tool and        an anatomical structure    -   wherein the identification of the deviation includes using a        machine learning model trained to identify deviations from        recommended sequences of events based on historical surgical        video footage, historical recommended sequences of events, and        information identifying deviations from the historical        recommended sequences of events in the historical video footage    -   wherein identifying the deviation includes comparing the        accessed frames to reference frames depicting the recommended        sequence of events    -   wherein outputting the predicted outcome includes updating an        electronic medical record associated with the patient    -   wherein outputting the predicted outcome includes transmitting        the predicted outcome to a data-receiving device associated with        a health care provider    -   determining at least one action likely to improve the predicted        outcome based on the accessed frames    -   providing a recommendation based on the determined at least one        action

Systems and methods disclosed herein involve unconventional improvementsover conventional approaches. Descriptions of the disclosed embodimentsare not exhaustive and are not limited to the precise forms orembodiments disclosed. Modifications and adaptations of the embodimentswill be apparent from consideration of the specification and practice ofthe disclosed embodiments. Additionally, the disclosed embodiments arenot limited to the examples discussed herein.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include hardware and software, but systems and methodsconsistent with the present disclosure may be implemented as hardwarealone.

Computer programs based on the written description and methods of thisspecification are within the skill of a software developer. The variousfunctions, scripts, programs, or modules may be created using a varietyof programming techniques. For example, programs, scripts, functions,program sections or program modules may be designed in or by means oflanguages, including JAVASCRIPT, C, C++, JAVA, PHP, PYTHON, RUBY, PERL,BASH, or other programming or scripting languages. One or more of suchsoftware sections or modules may be integrated into a computer system,non-transitory computer readable media, or existing communicationssoftware. The programs, modules, or code may also be implemented orreplicated as firmware or circuit logic.

Moreover, while illustrative embodiments have been described herein, thescope may include any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication, which examples are to be construed as non-exclusive.Further, the steps of the disclosed methods may be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the specification and examples beconsidered as exemplary only, with a true scope and spirit beingindicated by the following claims and their full scope of equivalents.

What is claimed is:
 1. A non-transitory computer readable mediumcontaining instructions that, when executed by at least one processor,cause the at least one processor to execute operations enablingdetermination and notification of an omitted event in a surgicalprocedure, the operations comprising: accessing frames of video capturedduring a specific surgical procedure; accessing stored data identifyinga recommended sequence of events for the surgical procedure; comparingthe accessed frames with the recommended sequence of events to identifyan indication of a deviation between the specific surgical procedure andthe recommended sequence of events for the surgical procedure, whereincomparing includes using a machine learning model to analyze theaccessed frames and identify the indication of the deviation between thespecific surgical procedure and the recommended sequence of events forthe surgical procedure; determining a name of an intraoperative surgicalevent associated with the deviation; and providing a notification of thedeviation including the name of the intraoperative surgical eventassociated with the deviation.
 2. The non-transitory computer readablemedium of claim 1, wherein identifying the indication of the deviationand providing the notification occurs in real time during the surgicalprocedure.
 3. The non-transitory computer readable medium of claim 1,wherein the operations further comprise: receiving an indication that aparticular action is about to occur in the specific surgical procedure;identifying, using the recommended sequence of events, a preliminaryaction to the particular action; determining, based on an analysis ofthe accessed frames, that the identified preliminary action did not yetoccurred; in response to the determination that the identifiedpreliminary action did not yet occurred, identifying the indication ofthe deviation.
 4. The non-transitory computer readable medium of claim3, wherein the indication that the particular action is about to occuris an entrance of a particular medical instrument to a selected regionof interest.
 5. The non-transitory computer readable medium of claim 1,wherein the specific surgical procedure is a cholecystectomy.
 6. Thenon-transitory computer readable medium of claim 1, wherein therecommended sequence of events is based on a critical view of safety. 7.The non-transitory computer readable medium of claim 1, wherein thespecific surgical procedure is an appendectomy.
 8. The non-transitorycomputer readable medium of claim 1, wherein the specific surgicalprocedure is a hernia repair.
 9. The non-transitory computer readablemedium of claim 1, wherein the specific surgical procedure is ahysterectomy.
 10. The non-transitory computer readable medium of claim1, wherein the specific surgical procedure is a radical prostatectomy.11. The non-transitory computer readable medium of claim 1, wherein thespecific surgical procedure is a partial nephrectomy, and the deviationincludes neglecting to identify a renal hilum.
 12. The non-transitorycomputer readable medium of claim 1, wherein the specific surgicalprocedure is a thyroidectomy, and the deviation includes neglecting toidentify a recurrent laryngeal nerve.
 13. The non-transitory computerreadable medium of claim 1, wherein the operations further compriseidentifying a set of frames associated with the deviation, and whereinproviding the notification includes displaying the identified set offrames associated with the deviation.
 14. The non-transitory computerreadable medium of claim 1, wherein the indication that the particularaction is about to occur is based on an input from a surgeon performingthe specific surgical procedure.
 15. The non-transitory computerreadable medium of claim 1, wherein identifying the deviation comprisesdetermining that a surgical tool is in a particular anatomical region.16. The non-transitory computer readable medium of claim 1, wherein thespecific surgical procedure is a hemicolectomy.
 17. The non-transitorycomputer readable medium of claim 16, wherein the deviation includesneglecting to perform an anastomosis.
 18. The non-transitory computerreadable medium of claim 1, where identifying the indication of thedeviation is based on an elapsed time associated with an intraoperativesurgical procedure.
 19. A computer-implemented method for enablingdetermination and notification of an omitted event in a surgicalprocedure, the method comprising: accessing frames of video capturedduring a specific surgical procedure; accessing stored data identifyinga recommended sequence of events for the surgical procedure; comparingthe accessed frames with the recommended sequence of events to identifya deviation between the specific surgical procedure and the recommendedsequence of events for the surgical procedure, wherein comparingincludes using a machine learning model to analyze the accessed framesand identify the indication of the deviation between the specificsurgical procedure and the recommended sequence of events for thesurgical procedure; determining a name of an intraoperative surgicalevent associated with the deviation; and providing a notification of thedeviation including the name of the intraoperative surgical eventassociated with the deviation.
 20. A system for enabling determinationand notification of an omitted event in a surgical procedure, the systemcomprising: at least one processor configured to: access frames of videocaptured during a specific surgical procedure; access stored dataidentifying a recommended sequence of events for the surgical procedure;compare the accessed frames with the recommended sequence of events toidentify a deviation between the specific surgical procedure and therecommended sequence of events for the surgical procedure, whereincomparing includes using a machine learning model to analyze theaccessed frames and identify the indication of the deviation between thespecific surgical procedure and the recommended sequence of events forthe surgical procedure; determine a name of an intraoperative surgicalevent associated with the deviation; and provide a notification of thedeviation including the name of the intraoperative surgical eventassociated with the deviation.